2023-11-18 13:20:20,916 INFO StreamThr :60204 [internal.py:wandb_internal():86] W&B internal server running at pid: 60204, started at: 2023-11-18 13:20:20.914208 2023-11-18 13:20:20,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status 2023-11-18 13:20:20,922 INFO WriterThread:60204 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_132020-qg88991p/run-qg88991p.wandb 2023-11-18 13:20:20,923 DEBUG SenderThread:60204 [sender.py:send():380] send: header 2023-11-18 13:20:20,924 DEBUG SenderThread:60204 [sender.py:send():380] send: run 2023-11-18 13:20:21,360 INFO SenderThread:60204 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_132020-qg88991p/files 2023-11-18 13:20:21,369 INFO SenderThread:60204 [sender.py:_start_run_threads():1124] run started: qg88991p with start time 1700306420.91509 2023-11-18 13:20:21,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: check_version 2023-11-18 13:20:21,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: check_version 2023-11-18 13:20:21,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: run_start 2023-11-18 13:20:21,425 DEBUG HandlerThread:60204 [system_info.py:__init__():32] System info init 2023-11-18 13:20:21,425 DEBUG HandlerThread:60204 [system_info.py:__init__():47] System info init done 2023-11-18 13:20:21,425 INFO HandlerThread:60204 [system_monitor.py:start():194] Starting system monitor 2023-11-18 13:20:21,425 INFO SystemMonitor:60204 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-11-18 13:20:21,426 INFO HandlerThread:60204 [system_monitor.py:probe():214] Collecting system info 2023-11-18 13:20:21,426 INFO SystemMonitor:60204 [interfaces.py:start():190] Started cpu monitoring 2023-11-18 13:20:21,428 INFO SystemMonitor:60204 [interfaces.py:start():190] Started disk monitoring 2023-11-18 13:20:21,429 INFO SystemMonitor:60204 [interfaces.py:start():190] Started gpu monitoring 2023-11-18 13:20:21,430 INFO SystemMonitor:60204 [interfaces.py:start():190] Started memory monitoring 2023-11-18 13:20:21,431 INFO SystemMonitor:60204 [interfaces.py:start():190] Started network monitoring 2023-11-18 13:20:21,436 DEBUG HandlerThread:60204 [system_info.py:probe():196] Probing system 2023-11-18 13:20:21,544 DEBUG HandlerThread:60204 [gitlib.py:_init_repo():56] git repository is invalid 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_info.py:probe():244] Probing system done 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T11:20:21.437109', 'startedAt': '2023-11-18T11:20:20.879359', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--torch_compile', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub', '--push_to_hub_model_id=WhisperLargeFinnishV3'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.43611145019531}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} 2023-11-18 13:20:21,545 INFO HandlerThread:60204 [system_monitor.py:probe():224] Finished collecting system info 2023-11-18 13:20:21,545 INFO HandlerThread:60204 [system_monitor.py:probe():227] Publishing system info 2023-11-18 13:20:21,545 DEBUG HandlerThread:60204 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-11-18 13:20:21,548 DEBUG HandlerThread:60204 [system_info.py:_save_pip():68] Saving pip packages done 2023-11-18 13:20:21,548 DEBUG HandlerThread:60204 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-11-18 13:20:22,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/requirements.txt 2023-11-18 13:20:22,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/conda-environment.yaml 2023-11-18 13:20:23,606 DEBUG HandlerThread:60204 [system_info.py:_save_conda():87] Saving conda packages done 2023-11-18 13:20:23,609 INFO HandlerThread:60204 [system_monitor.py:probe():229] Finished publishing system info 2023-11-18 13:20:23,613 DEBUG SenderThread:60204 [sender.py:send():380] send: files 2023-11-18 13:20:23,613 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now 2023-11-18 13:20:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:23,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:23,834 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:23,835 DEBUG SenderThread:60204 [sender.py:send():380] send: config 2023-11-18 13:20:23,835 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:20:23,836 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:23,836 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:20:23,836 WARNING SenderThread:60204 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) 2023-11-18 13:20:23,837 DEBUG SenderThread:60204 [sender.py:send():380] send: telemetry 2023-11-18 13:20:24,224 INFO wandb-upload_0:60204 [upload_job.py:push():131] Uploaded file /tmp/tmphe77gjs0wandb/1613lfeo-wandb-metadata.json 2023-11-18 13:20:24,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/conda-environment.yaml 2023-11-18 13:20:24,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-metadata.json 2023-11-18 13:20:24,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:26,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:26,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:28,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:20:31,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:36,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:38,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:38,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:41,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:46,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:51,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:20:52,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 13:20:53,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:20:53,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:20:53,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:20:57,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:02,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:07,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:08,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:08,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:12,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:17,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:21,431 DEBUG SystemMonitor:60204 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-11-18 13:21:21,433 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:21:23,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:23,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:23,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:28,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:33,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:38,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:38,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:38,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:43,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:48,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:51,433 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:21:53,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:21:53,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:21:53,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:21:53,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:21:58,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:03,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:08,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:08,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:09,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:14,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:19,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:21,434 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:22:22,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:23,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:23,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:24,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:24,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:29,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:34,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:39,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:39,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:39,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 13:22:39,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:44,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:44,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:49,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:22:51,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:22:52,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:22:53,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:22:53,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:22:55,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:00,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:04,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:06,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:09,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:09,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:09,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:11,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:12,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:16,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:21,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:23:22,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:22,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:24,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:24,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:24,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:27,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:33,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:33,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:38,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:39,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:39,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:39,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:42,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:43,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:48,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:23:50,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:23:51,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:23:54,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:23:54,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:23:54,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:23:54,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:00,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:00,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:05,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:09,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:09,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:09,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:10,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:10,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:15,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:18,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:20,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:21,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:24:24,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:24,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:25,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:28,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:30,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:36,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:36,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:39,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:24:39,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:39,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:41,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:46,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:47,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:51,438 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:24:53,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:24:54,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:24:54,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:24:56,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:24:58,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:04,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:04,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:09,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:25:09,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:09,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:14,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:14,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:19,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:21,438 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:25:22,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:29,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:32,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:35,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:39,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:25:39,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:39,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:40,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:40,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:25:40,128 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 13:25:40,129 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:25:40,130 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:25:40,141 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:25:40,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:25:42,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:45,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:50,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:25:51,439 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:25:52,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:25:54,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:25:54,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:25:55,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:00,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:01,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:06,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:06,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 13:26:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:09,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:09,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:11,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:11,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:16,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:21,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:21,439 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:26:22,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:26,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:26,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:26,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:27,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:29,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:32,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:37,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:41,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:41,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:41,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:41,234 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:43,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:49,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:49,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:26:51,451 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:26:54,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:26:56,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:26:56,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:26:56,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:26:58,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:00,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:06,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:06,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:11,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:11,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:11,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:16,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:16,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:21,440 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:27:22,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:26,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:26,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:26,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:28,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:33,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:36,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:38,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:42,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:27:42,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:42,683 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:43,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:44,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:48,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:51,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:27:54,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:27:54,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:27:57,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:27:57,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:27:57,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:27:59,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:04,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:05,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:10,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:12,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:12,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:12,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:12,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:15,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:21,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:21,443 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:28:22,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:26,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:28,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:28,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:28,200 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:30,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:32,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:37,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:41,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:42,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:43,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:43,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:43,198 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:47,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:47,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:28:47,706 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:28:47,707 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:28:47,710 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:28:48,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:28:49,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:51,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:28:53,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:28:58,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:28:58,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:28:58,199 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:28:59,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:28:59,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:04,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:07,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:10,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:13,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:29:13,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:13,208 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:15,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:17,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:20,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:21,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:29:26,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:27,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:29:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:28,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:32,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:38,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:39,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:43,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:43,208 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:43,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:49,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:49,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:51,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:29:54,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:29:56,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:29:58,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:29:58,206 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:29:59,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:04,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:06,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:09,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:13,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:13,206 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:14,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:15,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:20,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:21,444 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:30:22,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:25,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:28,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:28,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:31,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:34,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:37,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:42,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:43,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:30:43,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:43,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:46,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:47,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:51,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:30:53,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:30:54,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:30:58,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:30:58,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:30:58,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:04,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:05,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:09,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:13,059 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:13,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:13,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:14,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:19,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:21,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:31:23,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:25,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:28,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:28,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:30,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:33,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:35,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:40,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:44,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:31:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:44,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:31:45,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:47,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:50,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:51,447 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:31:56,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:31:57,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:31:59,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:31:59,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:31:59,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:02,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:04,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:32:04,199 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:32:04,199 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:32:04,202 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:32:04,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:32:05,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:07,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:08,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:14,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:14,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:14,430 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:15,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:19,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:21,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:32:25,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:25,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:29,430 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:30,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:35,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:35,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:40,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:43,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:44,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:44,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:32:46,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:51,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:32:52,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:53,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:32:57,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:32:59,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:32:59,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:32:59,432 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:00,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:02,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:07,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:10,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:13,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:14,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:14,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:18,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:20,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:21,449 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:33:24,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:28,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:29,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:29,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:34,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:39,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:40,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:45,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:33:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:33:45,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:33:49,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:33:50,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:51,449 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:33:55,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:33:57,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:00,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:00,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:00,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:01,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:07,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:07,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:12,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:15,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:15,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:15,487 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:17,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:17,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:21,452 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:34:23,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:27,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:28,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:30,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:34:30,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:30,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:33,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:35,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:38,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:43,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:34:45,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:34:45,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:34:49,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:51,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:34:55,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:34:55,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:00,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:01,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:35:01,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:01,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:03,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:06,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:11,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:35:11,141 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:35:11,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:35:11,144 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:35:11,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:35:12,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:13,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:16,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:16,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:17,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:21,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:35:21,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:22,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:27,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:31,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:31,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:31,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:33,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:38,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:41,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:44,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:47,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:35:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:35:47,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:35:49,007 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:35:49,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:51,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:35:54,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:35:59,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:00,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:02,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:02,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:06,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:07,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:11,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:16,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:17,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:17,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:19,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:21,454 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:36:21,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:26,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:27,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:31,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:32,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:32,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:37,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:42,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:43,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:36:47,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:36:48,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:51,455 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:36:53,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:36:53,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:36:58,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:01,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:02,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:02,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:04,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:10,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:11,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:15,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:17,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:17,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:37:17,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:21,455 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:37:21,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:31,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:32,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:32,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:37:32,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:32,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:37,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:39,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:42,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:47,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:37:47,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:37:48,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:49,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:51,456 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:37:53,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:37:57,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:37:59,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:02,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:38:02,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:02,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:04,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:07,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:09,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:14,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:17,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:17,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:18,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:20,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:21,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:38:24,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:38:24,186 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:38:24,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:38:24,188 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:38:25,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:38:25,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:26,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:26,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:31,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:32,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:32,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:36,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:36,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:41,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:44,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:46,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:47,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:38:47,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:38:51,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:38:52,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:38:54,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:38:57,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:02,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:02,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:03,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:08,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:13,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:14,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:17,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:17,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:19,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:21,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:21,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:39:24,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:30,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:31,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:32,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:32,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:35,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:40,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:41,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:45,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:47,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:39:47,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:39:47,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:39:51,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:51,459 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:39:51,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:39:56,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:39:59,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:02,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:02,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:02,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:08,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:09,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:13,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:17,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:17,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:17,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:18,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:21,459 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:40:23,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:25,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:28,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:33,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:34,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:34,188 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:37,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:39,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:45,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:45,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:40:49,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:40:49,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:40:50,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:51,461 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:40:55,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:40:56,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:00,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:04,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:04,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:04,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:06,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:11,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:14,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:16,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:19,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:19,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:21,462 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:41:22,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:24,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:27,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:31,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:41:31,482 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:41:31,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:41:31,484 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:41:32,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:41:33,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:33,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:34,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:34,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:34,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:39,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:42,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:45,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:49,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:41:49,188 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:41:50,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:51,462 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:41:51,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:41:55,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:41:59,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:00,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:04,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:04,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:06,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:11,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:15,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:17,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:19,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:19,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:21,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:42:22,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:23,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:28,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:33,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:34,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:34,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:34,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:43,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:44,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:48,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:49,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:42:49,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:42:49,998 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:42:51,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:42:52,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:42:54,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:42:59,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:04,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:04,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:04,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:04,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:04,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:10,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:12,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:15,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:19,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:19,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:19,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:21,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:21,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:43:22,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:26,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:30,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:31,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:36,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:36,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:36,272 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:37,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:40,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:42,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:47,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:51,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:43:51,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:43:51,273 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:43:51,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:43:52,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:43:54,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:43:57,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:01,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:03,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:06,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:06,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:06,274 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:08,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:09,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:14,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:19,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:19,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:21,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:21,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:21,272 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:21,596 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:44:25,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:27,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:30,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:35,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:37,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:37,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:37,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:39,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:40,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:45,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:47,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:44:47,713 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:44:47,713 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:44:47,716 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:44:47,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:44:49,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:44:51,466 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:44:51,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:52,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:44:52,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:44:52,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:44:56,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:44:58,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:01,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:06,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:07,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:45:07,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:07,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:08,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:11,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:17,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:18,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:21,466 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:45:22,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:22,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:45:22,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:22,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:27,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:28,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:32,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:37,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:37,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:37,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:38,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:42,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:47,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:50,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:45:51,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:45:52,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:45:52,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:45:53,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:45:59,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:00,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:04,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:07,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:46:07,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:07,719 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:09,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:10,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:14,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:18,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:20,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:21,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:46:22,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:22,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:25,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:29,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:31,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:36,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:37,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:37,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:37,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:41,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:46,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:47,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:51,468 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:46:52,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:46:53,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:46:53,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:46:53,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:46:55,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:46:57,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:02,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:06,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:07,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:08,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:08,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:13,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:16,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:19,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:47:23,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:23,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:23,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:24,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:24,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:29,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:32,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:35,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:39,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:39,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:39,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:41,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:42,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:46,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:50,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:47:51,469 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:47:51,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:47:54,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:47:54,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:47:54,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:47:57,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:00,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:48:00,576 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:48:00,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:48:00,579 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:48:01,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:48:02,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:02,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:07,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:09,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:48:09,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:09,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:10,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:13,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:18,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:20,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:21,470 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:48:24,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:24,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:24,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:28,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:29,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:34,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:38,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:39,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:39,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:40,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:45,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:46,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:48:50,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:51,471 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:48:55,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:48:55,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:48:55,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:48:55,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:48:58,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:01,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:06,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:07,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:10,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:10,393 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:12,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:16,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:18,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:21,472 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:49:23,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:24,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:25,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:25,384 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:28,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:34,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:34,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:39,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:40,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:49:40,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:40,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:49:44,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:45,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:50,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:51,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:49:54,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:49:55,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:49:55,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:49:55,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:00,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:02,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:05,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:10,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:10,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:11,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:12,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:16,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:20,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:21,473 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:50:22,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:25,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:25,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:27,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:30,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:33,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:38,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:40,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:40,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:40,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:43,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:48,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:50:49,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:51,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:50:55,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:50:55,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:50:55,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:50:55,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:50:58,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:01,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:05,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:51:05,402 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:51:05,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:51:05,406 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:51:05,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:51:06,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:07,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:07,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:10,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:10,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:12,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:17,041 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:18,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:21,475 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:51:24,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:25,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:51:25,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:25,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:27,095 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:29,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:34,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:35,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:39,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:51:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:41,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:51:45,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:45,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:50,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:51,476 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:51:52,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:51:55,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:51:56,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:51:56,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:01,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:02,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:06,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:11,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:11,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:12,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:16,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:17,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:21,477 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:52:23,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:24,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:26,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:26,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:29,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:34,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:35,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:40,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:41,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:41,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:42,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:46,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:51,477 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:52:51,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:52:52,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:52:56,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:52:56,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:52:56,236 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:52:56,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:00,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:01,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:06,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:10,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:11,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:11,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:12,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:17,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:20,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:21,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:53:22,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:26,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:26,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:53:26,239 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:28,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:28,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:33,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:38,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:38,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:41,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:41,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:43,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:46,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:49,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:51,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:53:54,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:53:56,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:53:56,235 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:53:57,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:53:59,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:04,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:07,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:09,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:12,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:12,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:13,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:54:13,506 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:54:13,507 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:54:13,509 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:54:14,242 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:54:14,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:15,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:15,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:20,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:54:25,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:26,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:27,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:27,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:31,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:33,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:36,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:42,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:42,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:42,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:47,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:51,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:54:52,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:54:52,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:54:57,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:54:57,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:54:57,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:02,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:02,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:07,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:10,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:12,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:12,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:13,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:18,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:20,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:21,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:55:24,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:27,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:27,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:28,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:29,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:34,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:38,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:39,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:42,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:55:45,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:48,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:51,481 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:55:51,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:56,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:55:56,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:55:57,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:55:57,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:02,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:07,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:07,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:12,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:56:12,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:12,460 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:13,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:15,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:18,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:21,482 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:56:24,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:25,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:27,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:27,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:29,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:35,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:35,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:40,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:42,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:42,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:56:45,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:45,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:50,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:51,483 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:56:55,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:56:56,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:56:57,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:56:57,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:02,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:03,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:07,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:12,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:12,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:12,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:13,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:17,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:20,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 13:57:20,739 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 13:57:20,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 13:57:20,742 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 13:57:21,483 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:57:21,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 13:57:22,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:23,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:28,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:28,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:28,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:28,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:32,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:34,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:40,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:40,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:43,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:43,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:43,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:57:46,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:48,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:51,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:51,484 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:57:56,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:57:58,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:57:58,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:57:58,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:57:58,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:02,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:07,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:08,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:12,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:15,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:58:15,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:15,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:18,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:19,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:21,484 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:58:23,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:27,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:28,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:30,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:30,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:33,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:37,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:39,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:45,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:45,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:58:45,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:45,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:58:50,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:58:51,485 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:58:55,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:58:56,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:00,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:00,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 13:59:00,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:01,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:03,301 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:06,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:11,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:13,367 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:15,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:15,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:16,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:21,485 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:59:22,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:23,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:27,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:30,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:30,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:32,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:35,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:38,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:43,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:45,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 13:59:45,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 13:59:47,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:48,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:51,486 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 13:59:53,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 13:59:55,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 13:59:58,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:01,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:00:01,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:01,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:04,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:05,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:09,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:14,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:15,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:16,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:16,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:20,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:21,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:00:23,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:25,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:30,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:31,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:00:31,106 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:00:31,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:00:31,109 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:00:31,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:31,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:31,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:00:32,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:36,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:41,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:42,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:46,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:00:46,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:00:47,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:51,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:00:51,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:00:53,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:00:58,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:01,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:01,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:01,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:03,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:08,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:11,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:14,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:17,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:17,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:17,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:19,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:19,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:21,488 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:01:24,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:29,277 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:30,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:32,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:32,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:32,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:35,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:37,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:41,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:47,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:01:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:01:47,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:01:47,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:51,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:01:52,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:01:55,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:01:58,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:03,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:03,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:03,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:02:03,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:05,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:08,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:14,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:15,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:18,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:18,355 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:19,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:21,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:02:23,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:25,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:30,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:33,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:33,358 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:33,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:35,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:40,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:41,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:45,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:49,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:02:49,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:02:49,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:02:51,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:02:51,490 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:02:51,841 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:02:56,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:01,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:01,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:04,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:04,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:04,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:07,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:09,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:12,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:18,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:19,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:19,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:19,233 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:20,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:21,490 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:03:23,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:28,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:29,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:34,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:35,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:03:35,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:35,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:36,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:03:36,575 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:03:36,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:03:36,578 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:03:37,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:03:37,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:39,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:39,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:44,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:47,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:49,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:03:50,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:03:50,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:03:51,492 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:03:55,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:03:55,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:00,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:05,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:05,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:05,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:06,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:11,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:15,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:16,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:20,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:20,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:04:20,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:21,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:04:22,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:25,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:28,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:33,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:35,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:35,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:35,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:38,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:44,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:45,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:49,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:04:50,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:04:50,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:04:50,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:04:51,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:04:53,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:04:55,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:00,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:03,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:05,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:05,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:05,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:11,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:11,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:16,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:20,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:20,990 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:21,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:05:21,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:21,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:26,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:29,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:32,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:37,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:37,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:05:37,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:37,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:39,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:42,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:48,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:49,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:51,494 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:05:52,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:05:52,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:05:52,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:05:53,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:05:58,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:05:59,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:04,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:07,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:07,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:07,421 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:09,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:10,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:14,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:18,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:20,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:21,495 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:06:22,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:22,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:22,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:26,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:28,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:31,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:36,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:37,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:37,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:38,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:42,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:44,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:06:44,758 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:06:44,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:06:44,761 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:06:45,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:06:45,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:47,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:47,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:06:52,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:06:52,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:06:52,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:06:53,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:06:55,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:06:58,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:03,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:03,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:07:07,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:09,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:13,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:15,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:20,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:21,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:07:22,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:22,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:23,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:25,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:30,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:33,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:36,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:07:37,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:37,436 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:41,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:43,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:47,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:07:52,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:07:52,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:07:52,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:07:53,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:07:57,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:02,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:03,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:07,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:07,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:08,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:11,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:14,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:20,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:21,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:08:22,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:08:22,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:25,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:30,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:32,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:35,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:37,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:37,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:08:41,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:44,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:47,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:51,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:08:52,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:08:52,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:08:52,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:08:52,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:08:57,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:02,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:04,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:07,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:07,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:07,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:12,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:14,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:17,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:21,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:09:22,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:22,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:23,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:24,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:28,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:33,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:34,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:37,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:37,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:39,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:43,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:45,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:50,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:09:51,500 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:09:52,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:09:52,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:09:53,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:09:55,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:00,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:01,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:10:01,036 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:10:01,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:10:01,039 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:10:01,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:10:01,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:03,778 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:06,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:08,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:08,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:08,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:11,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:11,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:16,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:21,501 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:10:23,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:23,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:23,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:23,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:26,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:31,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:33,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:36,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:38,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:38,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:38,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:42,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:51,502 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:10:52,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:10:53,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:10:53,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:10:53,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:10:54,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:10:57,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:02,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:04,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:08,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:08,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:08,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:08,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:13,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:14,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:18,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:21,503 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:11:23,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:23,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:23,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:23,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:26,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:28,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:34,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:34,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:38,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:38,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:40,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:45,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:46,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:11:50,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:51,503 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:11:55,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:11:55,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:11:55,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:11:56,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:11:58,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:01,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:06,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:06,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:10,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:10,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:12,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:14,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:21,504 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:12:24,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:24,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:25,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:25,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:25,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:29,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:34,715 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:40,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:40,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:40,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:41,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:45,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:46,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:51,504 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:12:52,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:12:55,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:12:55,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:12:55,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:12:57,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:12:57,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:05,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:08,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:10,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:10,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:10,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:13,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:17,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:18,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:21,505 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:13:23,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:26,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:26,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:26,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:28,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:28,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:33,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:38,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:38,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:13:38,894 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:13:38,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:13:38,899 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:13:39,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:13:40,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:41,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:41,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:43,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:48,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:50,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:51,505 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:13:54,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:13:56,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:13:56,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:13:56,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:13:58,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:13:59,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:04,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:09,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:11,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:11,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:11,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:12,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:15,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:20,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:21,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:14:22,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:26,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:26,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:31,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:34,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:37,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:14:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:41,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:42,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:46,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:48,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:51,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:14:54,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:14:54,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:14:56,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:14:56,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:14:59,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:04,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:04,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:09,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:11,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:15:11,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:11,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:12,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:15,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:20,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:21,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:15:22,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:26,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:26,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:26,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:31,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:34,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:37,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:15:41,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:41,327 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:42,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:46,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:47,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:51,507 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:15:53,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:15:56,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:15:56,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:15:57,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:15:59,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:05,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:06,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:10,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:12,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:12,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:14,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:16,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:21,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:21,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:16:26,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:27,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:27,155 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:28,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:31,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:36,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:38,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:41,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:42,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:42,156 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:47,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:50,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:51,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:16:52,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:16:57,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:16:57,156 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:16:58,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:16:58,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:03,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:05,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:17:05,949 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:17:05,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:17:05,951 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:17:06,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:17:08,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:08,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:13,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:13,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:13,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:14,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:16,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:19,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:21,509 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:17:24,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:24,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:28,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:28,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:28,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:30,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:35,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:36,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:40,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:43,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:43,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:17:46,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:48,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:51,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:51,510 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:17:56,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:17:56,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:17:58,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:17:58,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:17:58,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:04,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:08,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:13,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:13,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:13,463 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:13,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:14,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:18,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:21,511 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:18:24,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:28,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:28,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:28,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:29,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:29,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:34,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:40,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:41,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:43,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:18:45,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:51,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:51,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:51,511 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:18:56,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:18:58,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:18:58,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:18:58,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:18:58,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:01,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:07,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:08,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:12,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:14,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:14,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:14,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:16,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:18,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:21,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:19:23,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:28,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:29,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:29,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:29,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:29,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:34,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:38,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:40,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:44,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:44,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:44,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:19:45,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:48,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:19:51,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:51,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:19:56,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:19:59,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:19:59,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:19:59,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:00,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:02,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:07,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:08,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:12,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:15,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:15,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:15,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:18,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:18,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:21,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:20:24,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:29,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:29,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:20:29,872 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:20:29,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:20:29,875 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:20:30,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:30,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:30,514 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:30,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:20:30,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:31,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:34,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:40,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:40,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:45,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:45,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:20:45,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:20:45,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:20:51,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:20:51,513 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:20:52,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:20:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:00,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:00,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:00,515 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:01,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:01,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:06,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:12,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:13,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:15,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:15,515 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:17,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:21,514 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:21:22,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:23,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:27,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:30,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:33,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:35,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:38,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:44,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:45,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:21:45,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:21:45,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:21:45,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:21:49,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:51,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:21:55,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:21:57,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:00,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:00,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:00,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:22:00,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:05,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:05,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:10,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:15,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:15,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:15,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:16,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:20,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:21,516 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:22:24,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:25,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:30,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:30,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:30,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:36,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:36,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:41,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:44,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:45,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:22:45,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:22:46,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:51,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:22:52,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:22:52,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:22:57,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:00,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:00,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:02,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:06,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:08,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:13,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:15,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:15,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:16,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:18,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:21,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:23:24,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:26,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:30,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:30,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:30,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:35,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:39,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:41,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:45,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:23:45,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:23:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:49,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:23:51,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:23:52,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:57,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:23:58,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:23:58,201 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:23:58,201 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:23:58,203 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:23:59,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:23:59,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:00,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:00,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:00,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:02,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:07,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:09,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:13,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:15,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:15,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:15,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:19,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:19,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:21,518 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:24:24,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:27,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:29,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:30,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:30,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:30,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:35,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:37,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:40,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:45,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:45,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:24:45,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:24:45,918 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:24:47,548 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:24:51,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:51,518 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:24:56,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:24:59,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:00,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:00,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:02,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:07,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:11,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:12,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:15,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:15,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:18,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:20,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:21,519 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:25:23,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:28,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:30,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:30,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:25:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:34,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:38,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:39,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:44,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:46,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:25:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:25:46,655 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:25:48,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:25:49,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:51,519 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:25:55,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:25:59,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:01,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:01,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:01,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:01,655 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:06,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:11,095 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:12,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:17,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:18,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:18,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:21,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:21,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:26:23,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:28,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:29,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:33,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:33,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:33,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:34,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:37,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:39,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:44,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:48,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:26:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:26:48,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:26:49,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:49,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:26:51,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:26:54,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:26:59,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:01,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:03,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:03,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:03,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:05,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:09,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:10,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:15,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:18,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:18,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:19,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:27:19,016 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:27:19,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:27:19,018 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:27:19,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:27:20,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:21,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:21,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:27:21,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:26,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:29,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:31,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:33,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:33,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:33,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:37,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:41,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:42,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:47,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:27:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:27:48,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:27:49,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:27:51,521 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:27:53,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:58,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:27:59,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:03,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:03,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:03,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:04,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:09,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:10,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:14,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:18,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:18,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:18,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:20,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:21,521 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:28:25,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:27,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:31,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:33,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:33,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:33,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:36,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:37,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:41,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:46,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:47,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:48,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:28:48,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:28:48,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:28:51,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:28:52,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:28:57,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:28:58,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:03,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:03,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:03,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:03,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:08,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:13,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:13,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:18,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:18,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:18,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:19,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:21,522 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:29:23,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:24,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:29,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:33,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:33,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:33,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:35,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:35,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:40,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:46,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:47,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:48,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:29:48,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:29:48,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:29:51,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:29:51,523 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:29:55,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:29:56,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:01,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:03,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:03,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:03,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:03,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:07,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:12,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:13,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:17,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:18,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:18,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:18,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:21,524 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:30:22,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:23,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:27,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:32,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:33,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:30:33,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:33,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:35,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:38,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:43,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:45,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:30:45,787 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:30:45,788 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:30:45,791 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:30:45,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:30:46,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:47,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:48,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:30:48,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:30:48,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:51,525 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:30:53,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:30:56,016 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:30:58,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:03,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:03,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:04,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:06,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:09,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:15,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:16,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:18,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:18,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:21,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:21,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:31:25,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:26,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:31,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:33,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:31:33,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:33,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:35,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:36,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:41,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:47,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:47,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:48,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:31:48,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:31:48,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:31:51,526 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:31:52,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:31:57,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:31:58,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:03,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:03,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:03,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:03,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:05,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:08,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:13,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:14,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:18,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:18,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:18,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:19,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:21,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:32:25,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:25,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:30,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:34,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:32:34,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:35,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:37,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:40,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:45,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:46,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:49,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:32:49,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:32:51,527 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:32:52,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:32:55,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:32:57,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:02,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:04,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:04,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:05,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:07,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:13,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:14,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:18,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:19,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:19,714 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:21,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:33:23,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:26,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:29,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:34,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:34,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:36,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:40,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:44,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:45,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:49,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:33:49,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:33:50,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:33:51,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:33:54,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:33:56,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:01,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:01,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:34:01,633 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:34:01,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:34:01,636 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:34:02,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:34:03,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:04,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:04,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:04,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:06,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:11,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:13,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:17,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:19,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:19,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:21,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:34:23,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:23,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:28,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:33,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:34,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:34,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:35,539 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:38,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:44,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:45,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:34:49,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:34:49,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:34:49,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:51,531 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:34:55,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:34:59,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:01,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:04,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:04,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:06,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:09,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:12,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:17,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:19,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:19,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:19,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:21,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:35:23,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:29,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:29,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:34,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:35,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:35:35,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:35,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:37,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:39,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:44,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:47,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:35:50,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:50,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:35:50,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:35:50,675 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:35:51,534 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:35:55,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:35:58,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:01,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:06,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:06,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:06,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:06,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:08,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:11,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:16,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:18,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:21,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:21,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:21,049 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:21,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:21,533 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:36:26,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:31,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:32,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:36,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:36,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:36,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:37,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:42,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:44,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:47,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:51,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:36:52,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:36:52,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:36:52,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:36:53,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:36:54,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:36:58,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:03,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:04,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:07,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:07,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:07,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:08,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:12,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:14,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:19,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:21,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:37:22,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:22,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:22,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:22,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:24,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:29,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:33,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:37:33,214 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:37:33,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:37:33,217 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:37:33,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:37:34,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:35,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:35,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:37,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:37,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:37,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:40,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:45,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:45,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:50,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:37:51,536 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:37:52,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:37:52,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:37:52,410 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:37:53,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:37:56,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:01,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:03,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:07,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:07,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:07,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:11,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:13,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:18,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:21,536 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:38:21,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:22,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:38:22,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:22,411 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:23,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:28,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:33,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:33,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:37,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:37,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:39,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:44,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:38:44,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:49,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:51,537 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:38:52,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:38:52,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:38:52,544 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:38:55,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:38:56,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:00,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:04,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:05,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:07,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:07,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:11,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:14,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:16,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:21,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:39:22,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:22,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:22,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:26,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:28,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:33,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:36,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:37,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:37,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:38,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:43,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:48,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:39:48,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:51,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:39:52,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:39:52,543 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:39:54,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:39:58,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:00,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:05,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:07,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:40:07,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:07,596 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:08,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:10,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:15,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:18,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:20,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:21,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:40:22,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:22,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:26,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:31,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:32,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:36,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:37,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:37,543 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:42,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:42,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:47,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:51,539 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:40:52,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:40:52,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:40:52,545 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:40:52,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:40:57,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:02,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:41:02,143 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:41:02,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:41:02,146 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:41:02,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:41:03,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:03,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:07,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:07,544 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:09,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:14,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:15,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:20,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:21,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:41:22,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:22,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:25,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:26,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:30,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:35,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:37,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:37,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:38,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:40,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:45,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:46,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:41:50,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:51,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:41:54,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:41:54,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:41:54,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:41:55,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:41:58,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:00,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:05,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:08,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:09,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:09,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:09,656 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:10,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:15,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:21,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:42:21,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:22,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:24,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:24,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:24,657 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:26,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:32,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:34,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:37,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:39,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:39,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:42,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:43,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:48,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:51,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:42:53,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:42:54,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:42:54,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:42:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:42:54,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:42:59,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:02,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:04,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:09,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:09,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:09,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:09,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:12,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:15,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:21,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:21,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:21,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:43:24,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:24,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:24,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:26,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:30,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:31,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:36,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:39,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:39,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:42,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:42,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:47,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:51,543 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:43:53,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:43:54,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:43:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:43:54,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:43:54,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:43:58,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:02,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:03,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:08,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:09,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:09,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:14,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:16,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:20,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:21,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:44:24,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:24,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:24,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:25,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:26,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:30,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:35,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:37,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:44:37,036 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:44:37,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:44:37,039 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:44:37,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:44:38,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:39,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:39,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:39,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:39,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:40,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:45,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:50,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:44:51,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:44:51,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:44:54,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:44:54,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:44:54,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:44:56,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:00,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:02,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:07,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:09,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:09,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:10,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:12,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:17,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:21,545 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:45:22,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:22,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:24,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:24,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:24,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:28,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:33,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:34,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:38,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:40,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:45:40,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:40,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:45:44,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:44,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:49,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:51,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:45:54,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:45:54,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:45:55,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:45:55,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:00,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:05,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:05,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:10,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:10,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:10,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:16,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:17,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:21,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:21,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:46:25,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:25,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:27,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:27,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:32,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:35,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:37,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:41,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:46:41,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:41,276 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:43,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:45,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:48,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:51,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:46:52,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:46:53,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:46:56,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:46:56,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:46:56,275 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:46:59,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:04,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:04,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:09,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:12,813 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:14,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:15,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:20,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:21,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:47:24,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:26,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:27,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:27,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:27,811 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:32,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:32,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:37,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:42,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:42,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:42,812 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:42,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:47:43,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:48,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:51,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:47:52,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:47:53,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:47:57,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:47:57,812 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:47:58,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:03,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:48:03,365 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:48:03,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:48:03,369 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:48:03,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:48:04,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:04,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:05,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:09,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:13,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:13,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:13,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:14,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:16,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:19,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:21,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:48:24,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:25,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:28,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:28,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:28,715 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:30,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:35,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:36,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:41,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:43,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:43,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:43,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:48:45,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:46,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:51,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:48:51,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:53,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:48:56,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:48:58,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:48:58,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:48:58,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:02,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:05,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:08,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:13,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:13,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:13,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:13,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:15,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:18,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:21,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:49:23,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:24,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:29,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:29,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:29,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:29,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:31,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:34,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:39,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:43,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:44,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:49:44,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:44,011 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:49:45,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:50,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:51,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:49:52,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:49:55,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:49:59,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:49:59,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:00,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:02,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:05,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:10,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:14,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:14,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:14,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:15,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:20,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:21,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:50:22,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:26,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:29,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:50:29,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:29,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:31,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:37,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:38,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:42,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:44,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:44,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:50:44,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:46,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:48,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:51,551 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:50:53,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:58,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:50:58,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:50:59,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:50:59,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:50:59,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:51:03,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:09,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:11,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:14,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:14,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:14,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:19,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:20,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:21,552 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:51:26,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:29,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:29,761 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:30,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:51:30,832 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:51:30,832 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:51:30,835 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:51:31,174 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:51:31,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:32,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:33,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:36,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:41,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:43,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:44,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:44,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:51:46,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:51,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:51:52,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:55,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:51:58,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:51:59,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:51:59,760 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:04,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:05,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:09,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:13,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:14,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:14,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:15,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:20,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:21,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:21,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:52:26,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:29,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:29,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:31,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:31,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:36,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:41,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:42,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:44,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:52:44,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:52:47,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:51,554 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:52:53,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:52:53,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:52:58,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:00,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:53:00,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:00,786 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:03,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:03,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:08,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:13,787 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:14,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:53:15,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:19,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:21,555 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:53:24,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:25,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:30,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:30,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:30,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:32,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:36,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:41,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:42,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:45,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:53:45,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:53:47,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:51,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:53:51,555 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:53:52,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:53:57,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:00,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:00,786 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:01,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:02,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:07,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:11,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:13,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:15,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:18,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:21,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:21,556 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:54:24,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:29,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:30,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:54:30,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:30,851 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:33,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:35,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:40,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:45,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:45,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:45,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:54:45,849 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:54:51,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:51,556 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:54:54,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:54:54,940 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:54:54,941 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:54:54,944 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:54:55,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:54:56,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:54:56,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:54:57,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:00,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:00,849 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:02,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:07,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:07,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:12,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:16,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:55:16,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:16,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:18,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:19,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:21,557 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:55:23,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:28,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:31,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:31,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:33,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:33,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:38,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:41,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:44,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:46,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:55:46,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:55:46,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:55:49,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:51,557 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:55:51,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:55:55,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:55:59,836 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:00,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:01,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:01,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:01,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:06,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:11,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:11,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:16,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:16,138 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:17,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:21,558 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:56:21,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:22,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:27,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:32,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:32,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:32,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:33,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:35,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:39,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:44,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:47,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:56:47,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:56:47,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:56:47,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:56:49,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:51,561 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:56:55,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:56:55,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:00,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:02,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:02,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:02,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:05,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:07,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:10,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:15,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:16,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:17,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:17,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:21,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:57:21,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:25,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:27,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:32,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:32,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:32,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:35,509 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:37,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:43,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:43,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:47,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:57:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:57:47,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:57:48,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:51,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:57:54,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:57:55,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:57:59,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:02,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:03,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:04,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:09,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:13,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:15,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:17,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:20,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:21,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 14:58:21,450 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 14:58:21,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 14:58:21,453 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 14:58:21,563 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:58:21,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 14:58:22,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:23,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:26,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:31,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:32,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:32,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:32,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:37,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:43,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:43,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:47,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:58:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:58:47,664 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:58:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:51,563 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:58:52,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:58:54,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:58:59,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:00,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:02,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:02,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:04,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:07,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:08,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:10,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:15,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:16,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:17,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:17,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:17,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:20,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:21,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:59:26,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:31,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:32,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:32,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:32,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:37,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:41,311 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:42,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:47,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 14:59:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 14:59:47,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 14:59:48,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:49,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 14:59:51,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 14:59:54,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 14:59:59,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:01,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:02,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:02,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:04,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:09,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:13,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:15,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:17,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:17,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:17,662 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:20,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:21,565 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:00:23,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:26,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:31,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:32,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:32,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:33,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:36,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:41,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:45,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:00:47,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:47,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:00:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:00:47,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:00:51,566 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:00:52,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:57,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:00:57,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:02,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:02,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:07,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:08,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:13,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:15,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:17,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:17,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:17,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:18,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:21,566 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:01:24,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:25,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:29,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:34,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:34,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:39,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:43,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:01:43,791 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:01:43,792 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:01:43,794 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:01:44,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:01:45,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:45,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:46,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:47,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:01:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:01:47,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:01:50,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:01:51,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:01:54,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:01:56,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:01,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:02,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:02,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:04,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:06,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:12,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:14,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:17,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:17,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:17,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:17,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:21,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:02:22,341 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:22,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:27,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:32,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:32,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:34,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:38,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:43,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:44,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:47,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:02:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:02:47,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:02:48,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:51,568 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:02:54,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:02:55,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:02:59,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:02,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:02,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:02,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:03,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:05,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:10,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:13,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:15,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:17,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:17,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:17,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:21,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:21,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:03:21,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:26,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:31,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:32,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:32,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:32,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:32,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:38,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:43,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:44,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:03:47,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:03:47,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:03:49,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:03:51,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:03:51,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:03:55,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:01,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:01,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:02,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:02,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:02,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:06,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:10,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:12,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:17,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:17,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:17,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:17,691 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:20,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:21,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:04:22,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:27,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:30,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:32,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:32,675 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:32,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:38,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:42,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:44,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:48,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:04:48,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:04:48,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:04:49,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:51,570 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:04:52,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:04:55,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:04:58,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:04:58,336 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:04:58,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:04:58,338 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:04:58,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:05:00,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:01,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:01,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:03,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:03,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:03,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:06,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:08,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:11,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:16,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:17,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:18,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:18,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:18,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:21,570 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:05:22,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:28,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:30,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:33,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:33,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:33,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:33,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:38,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:42,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:44,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:49,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:05:49,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:05:49,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:05:49,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:05:51,571 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:05:52,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:05:55,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:00,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:03,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:04,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:04,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:04,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:05,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:10,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:15,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:16,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:19,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:19,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:19,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:21,573 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:06:21,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:23,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:27,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:32,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:33,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:34,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:34,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:38,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:43,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:44,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:48,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:06:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:06:50,269 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:06:51,573 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:06:53,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:06:54,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:06:58,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:03,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:04,149 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:05,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:05,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:09,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:14,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:14,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:19,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:20,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:20,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:21,574 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:07:24,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:26,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:29,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:35,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:35,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:35,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:36,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:07:41,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:46,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:50,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:07:50,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:07:51,574 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:07:51,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:57,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:07:58,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:02,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:05,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:05,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:06,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:07,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:12,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:16,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:17,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:20,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:20,270 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:21,576 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:08:23,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:24,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:28,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:32,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:08:32,176 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:08:32,176 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:08:32,179 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:08:32,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:08:33,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:33,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:34,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:35,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:35,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:38,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:44,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:44,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:49,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:50,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:08:50,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:08:50,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:08:51,576 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:08:54,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:08:54,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:08:59,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:02,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:05,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:05,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:05,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:10,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:12,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:15,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:20,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:20,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:20,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:21,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:21,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:09:26,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:30,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:32,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:37,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:09:37,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:37,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:38,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:40,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:43,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:48,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:09:49,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:51,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:09:52,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:09:52,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:09:52,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:09:54,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:09:59,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:00,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:04,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:07,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:10:07,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:07,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:09,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:10,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:14,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:20,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:20,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:21,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:10:22,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:22,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:26,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:30,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:31,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:36,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:37,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:10:37,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:37,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:40,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:42,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:47,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:48,555 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:10:51,578 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:10:52,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:10:52,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:10:52,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:58,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:10:58,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:03,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:06,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:07,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:07,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:09,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:14,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:18,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:20,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:21,579 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:11:22,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:11:22,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:22,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:26,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:26,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:31,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:36,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:36,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:37,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:37,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:42,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:42,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:11:42,671 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:11:42,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:11:42,673 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:11:42,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:11:44,921 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:45,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:47,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:51,579 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:11:52,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:11:52,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:11:52,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:11:53,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:11:58,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:03,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:03,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:07,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:07,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:09,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:15,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:15,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:20,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:21,580 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:12:22,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:22,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:23,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:25,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:30,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:33,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:35,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:38,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:12:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:38,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:40,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:40,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:45,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:50,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:12:51,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:51,580 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:12:53,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:12:53,374 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:12:56,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:12:58,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:01,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:06,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:13:08,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:08,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:10,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:12,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:17,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:18,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:21,581 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:13:23,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:23,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:28,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:28,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:33,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:36,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:38,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:38,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:38,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:13:39,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:44,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:48,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:13:50,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:51,581 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:13:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:13:53,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:13:55,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:13:58,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:01,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:06,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:08,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:09,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:11,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:16,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:21,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:21,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:14:21,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:23,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:26,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:31,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:32,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:37,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:38,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:38,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:41,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:42,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:47,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:51,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:14:51,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:14:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:53,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:14:53,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:14:58,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:14:58,257 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:14:58,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:14:58,260 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:14:58,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:14:59,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:14:59,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:01,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:04,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:08,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:09,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:13,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:15,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:20,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:21,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:15:22,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:23,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:23,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:25,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:30,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:31,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:36,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:38,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:15:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:38,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:40,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:41,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:46,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:50,661 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:15:51,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:15:52,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:15:53,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:15:53,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:15:53,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:15:58,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:00,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:03,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:08,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:08,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:09,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:10,801 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:14,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:18,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:19,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:21,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:16:23,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:23,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:23,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:25,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:30,932 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:31,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:36,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:38,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:38,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:38,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:41,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:47,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:49,041 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:16:51,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:16:53,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:16:53,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:16:53,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:16:53,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:58,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:16:59,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:03,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:07,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:08,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:08,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:09,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:14,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:17,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:19,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:21,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:17:23,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:23,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:23,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:25,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:29,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:31,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:36,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:37,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:38,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:38,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:38,382 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:42,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:47,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:47,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:51,585 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:17:53,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:17:53,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:17:53,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:17:53,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:17:57,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:17:58,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:04,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:05,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:08,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:08,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:10,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:12,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:18:12,783 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:18:12,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:18:12,787 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:18:13,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:18:13,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:15,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:15,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:20,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:21,586 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:18:23,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:18:23,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:23,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:25,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:26,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:31,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:35,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:36,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:38,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:38,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:42,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:46,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:47,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:51,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:18:53,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:18:53,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:18:53,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:18:54,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:18:59,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:04,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:08,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:08,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:10,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:15,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:20,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:21,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:19:23,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:23,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:25,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:30,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:35,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:38,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:38,925 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:40,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:41,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:19:45,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:49,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:19:50,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:51,588 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:19:53,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:19:53,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:19:56,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:19:59,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:01,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:06,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:08,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:08,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:09,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:12,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:17,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:19,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:21,588 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:20:23,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:24,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:20:24,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:24,509 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:27,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:28,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:33,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:37,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:39,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:39,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:39,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:44,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:47,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:49,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:51,589 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:20:54,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:20:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:20:54,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:20:54,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:20:57,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:20:59,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:04,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:07,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:09,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:09,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:10,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:15,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:16,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:21,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:21,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:21:24,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:24,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:25,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:26,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:31,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:33,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:37,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:39,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:39,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:42,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:45,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:47,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:51,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:21:53,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:21:53,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:21:54,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:21:54,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:21:59,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:00,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:22:00,986 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:22:00,986 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:22:00,988 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:22:01,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:22:02,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:03,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:04,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:09,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:22:09,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:09,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:10,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:11,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:15,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:20,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:21,592 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:22:24,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:24,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:26,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:31,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:36,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:39,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:39,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:41,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:45,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:47,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:51,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:22:53,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:22:53,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:22:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:22:54,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:22:58,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:03,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:05,744 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:08,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:10,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:23:10,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:10,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:13,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:14,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:19,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:21,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:23:23,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:24,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:25,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:25,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:30,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:31,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:35,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:40,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:40,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:40,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:41,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:46,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:51,593 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:23:51,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:23:52,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:23:55,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:23:55,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:23:55,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:23:57,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:00,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:02,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:07,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:09,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:10,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:10,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:13,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:17,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:18,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:21,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:24:24,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:25,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:25,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:27,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:30,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:35,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:37,621 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:40,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:40,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:40,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:46,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:51,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:51,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:24:51,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:24:56,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:24:56,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:24:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:24:56,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:24:59,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:02,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:07,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:09,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:11,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:11,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:25:11,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:12,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:17,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:17,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:21,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:25:22,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:25:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:26,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:27,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:29,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:32,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:36,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:25:36,799 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:25:36,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:25:36,802 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:25:36,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:25:37,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:38,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:41,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:41,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:25:44,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:48,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:49,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:51,595 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:25:55,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:25:56,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:25:56,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:25:56,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:01,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:06,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:07,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:11,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:26:11,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:11,962 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:12,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:15,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:18,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:21,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:26:23,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:25,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:26,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:26,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:29,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:33,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:34,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:39,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:41,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:41,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:26:43,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:45,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:50,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:51,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:26:51,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:26:55,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:26:58,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:26:58,230 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:01,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:01,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:06,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:09,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:11,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:13,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:13,229 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:17,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:19,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:21,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:27:22,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:28,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:28,229 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:28,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:29,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:33,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:37,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:38,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:43,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:44,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:27:44,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:44,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:27:47,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:48,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:51,598 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:27:54,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:55,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:27:59,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:27:59,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:27:59,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:27:59,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:04,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:08,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:10,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:14,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:14,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:14,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:15,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:16,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:20,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:21,599 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:28:24,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:26,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:30,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:32,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:34,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:37,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:41,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:28:41,530 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:28:41,531 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:28:41,534 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:28:42,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:28:42,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:42,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:44,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:45,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:28:45,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:28:45,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:28:48,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:51,600 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:28:54,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:28:54,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:28:59,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:00,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:00,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:00,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:02,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:04,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:09,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:12,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:14,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:15,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:15,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:20,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:21,601 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:29:25,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:25,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:30,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:30,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:30,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:35,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:35,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:40,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:45,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:45,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:45,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:29:45,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:29:45,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:29:51,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:29:51,602 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:29:53,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:29:56,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:00,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:00,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:00,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:02,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:03,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:07,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:13,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:13,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:15,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:15,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:19,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:21,602 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:30:21,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:24,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:29,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:30,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:30,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:31,994 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:35,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:40,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:40,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:45,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:47,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:30:47,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:30:47,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:30:50,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:30:50,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:30:51,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:30:55,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:00,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:00,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:02,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:02,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:06,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:10,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:12,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:17,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:17,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:17,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:20,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:21,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:31:22,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:28,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:28,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:32,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:31:32,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:32,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:33,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:36,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:39,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:46,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:47,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:31:47,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:31:49,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:31:51,605 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:31:53,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:31:53,526 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:31:53,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:31:53,529 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:31:53,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:31:54,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:31:55,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:00,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:03,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:32:03,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:06,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:06,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:11,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:13,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:16,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:18,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:21,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:32:22,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:23,831 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:27,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:33,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:33,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:33,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:33,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:38,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:43,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:48,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:48,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:32:48,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:32:48,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:51,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:32:53,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:32:56,124 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:32:59,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:33:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:04,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:06,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:09,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:14,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:14,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:18,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:18,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:20,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:21,608 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:33:24,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:25,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:30,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:33,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:33,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:34,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:36,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:41,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:44,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:47,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:48,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:33:48,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:33:48,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:33:51,608 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:33:52,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:33:52,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:33:57,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:02,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:03,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:04,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:08,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:13,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:14,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:18,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:18,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:19,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:21,609 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:34:24,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:24,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:29,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:33,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:33,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:34,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:35,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:40,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:44,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:46,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:48,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:34:48,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:34:48,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:34:51,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:34:51,609 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:34:52,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:34:56,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:02,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:02,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:03,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:07,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:12,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:13,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:35:13,644 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:35:13,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:35:13,647 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:35:14,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:35:14,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:15,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:17,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:18,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:18,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:21,610 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:35:22,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:24,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:27,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:32,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:33,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:33,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:34,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:38,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:42,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:44,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:35:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:35:48,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:35:50,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:35:51,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:35:52,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:35:55,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:01,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:02,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:03,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:03,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:03,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:07,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:10,409 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:12,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:17,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:18,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:18,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:18,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:20,475 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:21,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:36:22,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:27,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:28,528 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:32,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:33,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:36:33,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:33,925 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:38,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:38,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:43,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:46,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:36:48,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:36:48,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:36:49,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:51,612 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:36:54,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:36:58,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:00,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:03,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:37:03,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:03,998 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:06,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:06,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:11,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:14,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:16,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:18,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:18,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:21,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:37:22,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:24,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:27,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:33,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:33,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:33,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:34,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:39,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:44,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:45,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:49,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:50,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:37:50,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:37:50,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:37:51,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:37:53,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:37:54,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:37:59,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:03,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:05,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:05,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:05,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:05,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:10,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:10,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:15,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:18,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:38:18,773 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:38:18,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:38:18,776 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:38:19,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:38:20,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:20,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:20,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:20,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:21,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:21,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:21,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:38:26,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:28,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:31,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:35,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:35,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:37,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:40,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:42,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:47,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:48,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:50,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:38:50,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:38:50,362 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:38:51,631 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:38:53,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:38:58,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:38:59,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:04,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:05,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:05,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:05,365 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:08,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:09,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:14,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:16,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:19,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:20,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:20,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:20,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:21,631 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:39:24,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:26,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:29,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:34,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:35,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:39:35,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:35,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:36,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:40,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:45,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:46,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:50,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:39:50,364 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:39:51,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:39:51,633 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:39:54,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:39:56,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:01,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:05,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:05,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:05,364 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:07,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:12,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:15,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:18,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:20,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:21,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:21,002 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:21,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:40:23,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:23,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:28,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:32,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:34,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:36,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:36,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:36,001 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:39,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:40,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:45,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:50,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:50,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:40:51,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:40:51,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:40:51,001 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:40:51,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:40:55,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:40:58,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:01,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:06,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:07,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:41:07,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:07,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:08,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:11,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:16,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:18,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:21,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:41:21,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:22,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:22,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:27,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:27,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:41:27,056 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:41:27,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:41:27,059 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:41:27,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:41:28,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:29,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:32,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:37,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:37,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:38,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:42,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:47,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:48,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:41:51,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:41:52,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:41:52,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:41:52,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:58,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:41:59,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:03,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:07,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:07,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:08,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:09,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:13,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:17,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:18,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:21,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:42:22,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:42:22,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:22,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:24,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:27,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:29,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:35,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:35,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:37,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:40,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:46,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:47,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:51,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:42:51,637 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:42:53,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:42:53,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:42:53,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:42:55,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:42:56,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:01,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:05,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:06,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:08,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:08,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:08,041 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:12,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:18,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:21,637 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:43:23,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:23,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:23,041 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:23,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:28,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:28,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:33,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:38,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:38,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:38,043 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:38,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:40,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:43,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:48,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:43:49,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:51,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:43:53,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:43:53,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:43:53,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:43:55,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:43:58,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:00,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:05,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:08,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:08,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:08,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:09,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:11,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:16,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:17,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:21,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:21,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:44:24,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:24,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:24,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:27,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:27,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:32,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:37,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:38,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:39,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:39,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:43,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:45,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:44:45,567 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:44:45,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:44:45,568 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:44:46,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:44:47,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:48,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:51,640 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:44:53,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:44:54,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:44:54,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:44:54,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:44:55,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:44:59,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:04,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:05,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:09,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:09,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:10,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:13,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:15,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:20,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:21,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:45:24,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:24,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:25,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:26,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:31,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:35,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:36,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:40,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:45:40,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:40,855 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:41,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:43,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:46,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:51,641 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:45:52,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:45:52,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:45:55,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:45:55,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:45:55,855 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:45:58,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:00,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:04,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:09,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:10,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:10,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:11,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:14,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:19,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:21,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:21,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:46:24,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:25,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:25,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:46:25,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:29,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:31,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:34,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:39,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:40,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:40,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:41,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:46,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:49,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:46:51,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:51,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:46:55,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:46:55,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:46:55,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:46:56,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:46:57,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:47:01,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:06,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:07,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:47:10,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:10,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:12,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:17,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:21,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:47:22,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:25,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:25,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:25,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:28,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:33,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:38,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:40,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:40,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:40,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:47:44,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:49,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:51,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:47:54,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:47:55,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:47:55,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:47:55,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:00,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:03,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:05,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:10,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:10,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:10,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:11,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:16,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:21,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:21,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:21,644 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:48:25,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:25,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:25,859 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:27,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:29,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:32,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:37,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:39,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:48:39,858 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:48:39,858 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:48:39,861 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:48:40,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:48:40,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:40,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:40,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:41,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:42,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:43,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:48,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:51,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:48:51,644 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:48:53,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:57,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:48:57,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:48:57,088 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:48:59,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:48:59,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:04,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:09,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:10,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:12,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:12,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:12,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:15,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:17,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:20,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:21,645 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:49:25,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:25,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:27,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:27,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:27,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:31,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:36,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:37,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:41,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:43,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:49:43,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:43,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:49:45,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:47,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:51,646 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:49:52,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:55,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:49:58,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:49:58,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:49:58,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:03,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:04,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:08,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:13,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:13,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:13,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:14,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:18,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:21,647 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:50:23,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:24,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:28,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:29,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:29,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:32,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:34,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:39,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:41,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:44,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:44,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:44,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:44,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:49,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:50:49,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:51,647 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:50:55,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:50:59,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:50:59,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:50:59,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:50:59,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:01,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:06,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:11,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:12,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:14,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:14,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:14,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:17,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:21,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:21,648 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:51:22,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:27,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:29,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:29,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:29,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:33,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:39,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:39,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:44,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:44,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:44,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:44,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:51:47,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:51:47,396 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:51:47,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:51:47,399 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:51:48,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:51:49,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:50,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:51:51,648 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:51:54,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:51:59,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:51:59,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:51:59,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:00,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:00,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:05,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:08,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:10,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:14,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:52:14,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:14,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:16,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:18,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:21,650 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:52:22,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:26,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:27,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:29,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:52:29,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:29,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:32,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:36,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:37,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:42,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:44,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:44,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:52:46,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:48,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:51,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:52:53,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:55,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:52:59,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:52:59,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:52:59,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:04,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:05,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:09,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:13,704 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:14,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:14,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:14,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:20,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:21,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:53:23,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:25,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:30,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:31,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:53:31,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:31,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:33,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:36,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:42,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:43,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:46,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:53:46,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:53:46,065 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:53:47,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:51,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:53:51,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:53:52,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:53:57,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:01,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:01,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:01,065 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:02,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:03,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:09,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:10,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:14,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:16,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:16,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:16,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:19,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:20,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:21,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:54:24,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:29,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:31,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:54:31,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:31,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:34,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:35,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:40,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:42,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:45,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:46,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:54:46,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:54:50,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:54:51,652 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:54:52,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:54:55,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:00,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:01,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:01,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:06,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:11,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:14,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:55:14,635 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:55:14,636 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:55:14,638 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:55:15,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:55:16,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:16,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:16,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:16,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:21,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:21,653 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:55:26,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:29,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:31,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:31,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:32,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:37,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:38,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:43,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:46,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:55:46,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:55:47,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:48,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:51,654 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:55:53,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:55:55,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:55:58,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:01,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:01,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:56:01,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:04,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:05,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:09,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:13,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:15,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:16,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:16,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:21,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:21,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:56:24,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:27,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:31,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:31,085 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:33,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:34,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:38,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:42,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:44,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:46,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:56:46,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:56:49,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:51,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:56:52,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:56:54,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:56:59,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:00,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:01,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:01,084 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:05,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:10,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:10,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:15,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:16,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:57:16,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:16,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:20,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:20,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:21,655 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:57:26,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:28,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:31,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:31,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:31,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:37,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:38,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:42,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:46,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:57:46,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:57:46,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:47,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:51,656 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:57:53,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:57:56,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:57:58,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:01,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:58:01,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:01,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:03,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:04,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:09,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:13,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:15,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:16,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:16,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:20,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:21,657 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:58:26,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:26,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 15:58:26,646 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 15:58:26,647 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 15:58:26,649 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 15:58:27,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 15:58:28,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:31,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:31,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:31,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:37,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:38,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:42,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:46,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:46,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:58:46,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:58:47,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:51,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:58:53,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:58:54,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:58:58,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:01,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:59:01,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:01,596 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:04,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:04,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:09,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:12,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:14,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:16,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:16,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:19,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:21,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:59:24,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:24,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:29,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:31,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:31,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:32,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:35,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:41,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:42,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:46,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:47,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 15:59:47,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 15:59:47,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 15:59:50,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 15:59:51,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 15:59:51,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 15:59:56,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:00,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:01,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:02,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:02,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:02,975 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:07,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:10,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:13,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:17,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:17,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:17,974 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:18,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:20,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:21,660 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:00:23,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:28,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:28,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:32,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:32,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:32,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:34,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:38,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:39,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:45,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:46,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:47,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:00:47,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:00:47,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:00:51,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:00:51,660 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:00:57,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:00:57,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:02,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:02,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:02,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:02,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:07,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:08,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:12,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:17,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:17,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:17,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:17,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:18,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:21,661 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:01:23,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:28,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:29,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:32,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:32,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:32,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:34,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:36,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:01:36,086 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:01:36,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:01:36,088 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:01:36,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:01:38,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:40,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:46,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:46,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:47,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:01:47,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:01:47,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:01:51,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:01:51,662 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:01:56,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:01:57,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:02,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:02,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:02,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:02,979 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:07,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:08,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:12,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:17,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:17,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:17,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:17,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:18,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:21,662 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:02:23,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:26,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:29,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:33,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:33,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:33,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:35,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:36,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:40,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:44,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:46,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:48,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:02:48,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:02:48,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:02:51,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:02:51,663 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:02:54,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:02:57,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:02,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:03,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:03,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:07,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:12,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:14,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:17,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:18,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:18,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:21,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:03:22,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:23,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:27,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:32,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:33,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:33,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:33,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:33,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:38,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:41,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:43,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:48,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:03:48,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:03:48,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:03:49,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:03:51,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:03:51,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:03:54,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:00,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:00,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:03,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:03,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:05,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:10,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:10,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:15,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:18,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:18,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:18,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:18,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:21,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:21,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:04:26,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:28,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:31,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:34,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:34,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:34,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:36,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:36,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:41,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:46,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:46,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:49,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:04:49,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:04:49,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:04:51,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:04:51,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:04:53,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:04:53,446 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:04:53,446 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:04:53,448 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:04:53,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:04:54,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:04:57,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:02,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:04,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:04,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:04,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:07,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:07,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:12,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:15,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:18,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:19,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:19,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:19,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:21,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:05:23,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:25,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:28,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:33,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:33,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:34,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:34,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:38,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:41,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:43,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:48,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:05:49,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:05:49,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:05:49,358 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:05:51,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:05:53,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:05:54,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:00,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:01,385 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:04,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:04,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:05,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:11,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:11,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:16,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:18,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:19,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:19,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:21,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:21,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:06:26,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:27,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:32,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:34,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:06:34,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:34,365 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:36,610 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:37,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:42,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:46,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:47,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:49,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:06:49,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:06:51,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:06:53,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:06:56,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:06:59,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:04,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:04,363 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:04,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:04,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:09,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:14,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:14,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:19,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:20,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:07:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:21,668 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:07:22,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:25,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:30,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:32,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:35,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:35,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:40,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:43,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:46,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:50,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:07:50,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:07:51,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:07:51,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:07:51,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:56,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:07:59,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:07:59,137 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:07:59,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:07:59,139 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:07:59,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:08:01,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:02,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:03,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:05,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:08:05,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:05,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:07,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:12,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:17,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:20,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:20,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:21,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:08:23,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:28,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:33,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:38,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:43,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:48,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:50,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:08:50,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:08:51,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:08:54,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:08:59,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:04,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:05,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:05,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:09,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:14,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:19,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:20,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:21,670 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:09:25,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:30,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:35,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:35,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:40,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:45,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:50,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:09:50,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:09:51,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:09:51,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:09:56,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:01,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:05,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:05,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:06,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:11,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:16,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:20,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:20,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:21,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:10:22,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:27,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:32,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:35,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:35,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:37,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:42,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:47,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:50,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:10:50,618 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:10:51,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:10:53,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:10:58,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:03,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:05,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:08,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:13,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:18,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:20,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:20,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:11:21,672 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:11:24,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:29,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:34,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:35,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:35,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:39,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:44,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:49,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:11:50,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:11:50,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:11:51,672 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:11:55,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:00,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:05,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:05,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:10,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:15,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:20,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:20,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:21,673 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:12:21,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:26,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:31,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:35,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:35,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:36,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:41,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:46,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:50,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:12:50,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:12:51,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:12:52,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:12:57,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:02,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:05,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:05,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:07,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:12,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:17,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:20,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:20,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:21,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:13:23,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:28,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:33,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:35,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:35,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:38,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:43,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:48,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:50,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:13:50,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:13:51,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:13:54,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:13:59,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:04,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:05,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:05,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:09,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:14,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:19,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:20,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:20,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:21,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:14:25,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:30,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:35,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:35,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:35,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:40,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:45,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:50,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:14:50,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:14:50,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:14:51,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:14:56,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:01,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:05,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:05,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:06,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:11,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:16,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:20,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:20,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:21,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:15:22,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:27,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:32,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:35,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:35,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:37,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:42,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:47,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:50,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:15:50,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:15:51,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:15:53,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:15:58,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:03,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:05,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:05,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:08,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:13,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:18,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:20,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:20,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:21,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:16:24,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:29,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:34,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:35,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:35,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:39,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:44,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:49,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:16:50,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:16:50,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:16:51,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:16:55,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:00,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:05,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:05,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:05,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:10,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:15,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:20,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:20,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:21,677 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:17:21,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:26,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:31,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:35,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:35,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:36,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:41,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:46,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:50,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:17:50,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:17:51,677 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:17:52,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:17:57,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:02,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:05,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:05,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:07,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:11,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:18:11,252 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,252 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,253 DEBUG SenderThread:60204 [sender.py:send():380] send: metric 2023-11-18 16:18:11,263 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:18:11,263 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:18:11,267 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:18:11,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:18:12,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:13,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:17,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:18,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/config.yaml 2023-11-18 16:18:20,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:18:20,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:20,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:21,678 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:18:23,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:28,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:33,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:36,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:18:36,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:18:36,820 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:18:38,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:44,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:44,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:46,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:18:49,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:51,687 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:18:55,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:18:57,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:19:00,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:05,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:10,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:15,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:20,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:21,679 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:19:26,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:31,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:36,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:41,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:46,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:46,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:19:46,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:19:51,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:19:52,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:19:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:19:54,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:19:54,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:19:55,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:19:57,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:19:58,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:03,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:08,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:09,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:13,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:15,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:15,810 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:17,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:20:19,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:21,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:20:24,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:24,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:29,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:35,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:39,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:39,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:40,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:46,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:51,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:20:52,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:20:54,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:20:54,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:20:54,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:20:57,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:03,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:08,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:09,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:09,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:09,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:21:13,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:13,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:19,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:21,705 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:21:23,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:24,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:24,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:26,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:31,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:35,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:36,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:39,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:39,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:21:39,745 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:42,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:45,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:47,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:51,710 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:21:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:21:54,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:21:54,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:21:55,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:21:58,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:04,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:06,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:09,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:09,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:09,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:14,297 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:15,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:20,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:21,711 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:22:24,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:24,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:24,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:25,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:31,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:36,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:38,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:39,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:39,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:41,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:46,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:47,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:51,712 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:22:53,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:22:54,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:22:54,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:22:56,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:22:58,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:04,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:05,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:09,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:09,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:23:09,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:13,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:15,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:21,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:21,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:23:23,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:24,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:24,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:26,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:31,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:33,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:37,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:39,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:23:39,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:39,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:43,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:44,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:49,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:51,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:23:54,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:23:54,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:23:54,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:23:54,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:23:59,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:02,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:05,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:09,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:10,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:12,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:15,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:20,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:21,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:24:22,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:24,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:24,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:25,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:30,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:31,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:24:31,149 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:24:31,149 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:24:31,152 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:24:31,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:24:32,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:36,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:39,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:39,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:41,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:42,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:46,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:51,715 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:24:52,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:24:52,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:24:54,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:24:54,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:24:57,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:00,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:03,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:09,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:09,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:10,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:15,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:18,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:20,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:21,715 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:25:25,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:25,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:26,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:28,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:31,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:36,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:40,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:40,180 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:40,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:25:42,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:42,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:47,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:51,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:25:51,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:25:52,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:25:55,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:25:55,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:25:58,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:03,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:05,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:08,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:10,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:10,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:13,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:14,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:18,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:21,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:26:24,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:24,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:25,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:25,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:30,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:34,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:36,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:41,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:41,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:26:41,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:41,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:44,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:46,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:51,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:26:51,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:26:52,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:26:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:26:56,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:26:56,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:26:56,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:01,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:02,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:07,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:11,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:11,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:11,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:12,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:12,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:17,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:20,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:21,717 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:27:23,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:26,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:26,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:28,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:30,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:34,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:39,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:40,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:41,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:41,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:41,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:45,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:47,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:27:47,828 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:27:47,829 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:27:47,831 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:27:48,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:27:48,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:50,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:27:51,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:51,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:27:56,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:27:56,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:27:56,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:27:56,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:27:58,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:01,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:06,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:07,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:11,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:11,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:11,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:12,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:17,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:19,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:21,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:28:23,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:26,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:26,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:27,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:28,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:34,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:34,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:39,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:41,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:41,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:41,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:44,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:45,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:51,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:28:51,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:28:52,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:28:56,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:28:56,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:28:56,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:28:56,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:00,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:01,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:07,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:11,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:11,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:11,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:12,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:12,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:17,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:21,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:29:22,438 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:23,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:26,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:26,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:26,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:29,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:32,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:35,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:40,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:41,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:41,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:41,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:29:42,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:45,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:50,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:29:51,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:51,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:29:56,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:29:57,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:29:57,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:29:57,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:00,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:02,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:08,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:10,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:12,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:12,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:13,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:18,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:18,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:21,722 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:30:23,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:27,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:27,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:28,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:29,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:34,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:36,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:39,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:42,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:30:42,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:42,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:30:44,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:47,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:49,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:51,724 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:30:54,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:30:54,126 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:30:54,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:30:54,128 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:30:55,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:30:55,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:55,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:30:56,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:30:57,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:30:57,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:00,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:06,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:07,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:11,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:12,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:12,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:12,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:15,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:17,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:21,725 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:31:22,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:25,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:27,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:27,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:27,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:27,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:33,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:34,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:39,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:42,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:42,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:42,586 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:31:44,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:44,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:49,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:51,725 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:31:54,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:31:55,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:31:57,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:31:57,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:31:57,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:00,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:04,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:06,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:11,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:12,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:12,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:12,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:12,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:16,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:21,726 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:32:22,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:22,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:27,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:28,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:28,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:28,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:30,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:33,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:39,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:40,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:43,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:43,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:43,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:44,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:49,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:50,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:32:51,726 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:32:55,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:32:58,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:32:58,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:32:58,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:32:58,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:01,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:07,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:09,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:12,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:14,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:33:14,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:33:14,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:33:17,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:21,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:21,727 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:33:22,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:27,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:29,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:33:29,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:33:29,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:33:33,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:33,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:39,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:43,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:44,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:33:44,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:33:44,495 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:33:44,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:49,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:51,727 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:33:53,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:33:54,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:33:59,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:33:59,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:33:59,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:34:00,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:01,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:06,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:09,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:34:09,956 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:34:09,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:34:09,960 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:34:10,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:34:10,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:11,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:11,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:14,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:34:14,495 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:34:17,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:21,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:21,727 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:34:22,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:27,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:30,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:34:30,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:34:30,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:34:32,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:33,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:38,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:42,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:43,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:45,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:34:45,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:34:49,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:34:50,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:34:51,728 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:34:54,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:00,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:00,786 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:00,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:35:00,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:35:05,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:08,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:11,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:16,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:16,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:35:16,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:35:16,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:35:18,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:21,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:21,729 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:35:26,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:31,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:31,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:35:31,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:35:31,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:35:32,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:37,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:41,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:42,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:46,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:35:46,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:35:46,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:35:47,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:49,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:35:51,729 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:35:53,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:58,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:35:59,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:01,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:36:01,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:36:04,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:09,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:09,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:14,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:16,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:36:16,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:36:19,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:20,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:21,730 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:36:25,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:29,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:31,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:31,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:36:31,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:36:36,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:39,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:41,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:46,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:36:46,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:36:47,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:47,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:51,730 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:36:52,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:36:57,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:36:58,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:01,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:37:01,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:37:04,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:07,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:09,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:14,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:16,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:37:16,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:37:17,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:20,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:21,731 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:37:25,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:25,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:37:25,847 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:37:25,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:37:25,849 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:37:26,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:37:26,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:30,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:31,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:37:31,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:37:36,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:36,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:41,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:44,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:37:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:37:47,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:37:47,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:51,732 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:37:52,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:37:55,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:37:57,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:02,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:38:02,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:38:02,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:38:02,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:05,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:08,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:13,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:13,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:17,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:38:17,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:38:17,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:38:19,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:21,750 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:38:23,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:24,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:29,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:32,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:38:32,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:38:32,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:38:33,389 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:35,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:40,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:43,449 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:45,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:38:47,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:38:47,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:38:51,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:38:51,736 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:38:53,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:38:56,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:01,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:01,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:02,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:39:02,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:39:02,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:39:07,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:11,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:12,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:17,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:39:17,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:39:17,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:39:18,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:21,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:21,744 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:39:23,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:28,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:31,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:32,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:39:32,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:39:32,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:39:34,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:40,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:41,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:45,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:47,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:39:47,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:39:47,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:39:48,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:39:50,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:39:51,734 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:39:55,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:00,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:02,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:40:02,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:40:03,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:06,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:11,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:11,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:16,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:17,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:40:17,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:40:21,735 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:40:22,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:23,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:28,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:31,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:32,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:40:32,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:40:33,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:38,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:40,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:40:40,156 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:40:40,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:40:40,159 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:40:40,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:40:41,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:44,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:48,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:40:48,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:40:48,290 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:40:49,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:51,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:40:51,735 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:40:54,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:40:59,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:00,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:03,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:41:03,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:41:03,289 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:41:05,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:10,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:11,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:15,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:18,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:41:18,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:41:18,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:41:21,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:21,681 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:21,735 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:41:26,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:29,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:31,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:33,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:41:33,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:41:33,289 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:41:37,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:39,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:42,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:47,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:48,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:41:48,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:41:48,667 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:41:51,736 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:41:51,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:41:52,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:41:58,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:00,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:03,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:42:03,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:42:03,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:42:03,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:09,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:10,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:14,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:18,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:42:18,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:42:18,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:42:19,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:20,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:21,737 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:42:25,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:29,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:31,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:33,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:42:33,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:42:33,615 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:42:36,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:37,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:42,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:47,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:48,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:42:48,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:42:48,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:42:51,737 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:42:52,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:42:53,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:42:58,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:03,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:03,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:43:03,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:43:03,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:43:03,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:08,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:13,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:14,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:19,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:19,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:43:19,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:43:19,667 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:43:21,738 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:43:21,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:24,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:30,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:32,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:34,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:43:34,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:43:34,666 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:43:35,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:40,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:41,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:46,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:49,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:43:49,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:43:49,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:43:51,738 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:43:51,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:52,184 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:43:57,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:43:59,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:43:59,457 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:43:59,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:43:59,459 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:44:00,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:44:01,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:02,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:02,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:04,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:44:04,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:44:04,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:44:07,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:10,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:13,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:18,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:19,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:19,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:44:19,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:44:19,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:44:21,739 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:44:23,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:29,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:29,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:34,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:35,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:44:35,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:44:35,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:44:37,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:39,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:45,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:49,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:44:50,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:44:50,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:44:50,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:44:50,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:51,739 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:44:55,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:44:57,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:01,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:05,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:45:05,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:45:05,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:45:05,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:07,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:13,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:15,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:19,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:20,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:45:20,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:45:20,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:45:21,741 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:45:23,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:24,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:30,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:33,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:35,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:45:35,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:45:35,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:45:35,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:40,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:43,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:45,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:50,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:45:51,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:45:51,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:45:51,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:45:51,742 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:45:54,083 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:45:56,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:01,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:02,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:06,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:46:06,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:46:06,018 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:46:07,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:12,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:12,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:17,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:21,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:46:21,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:46:21,018 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:46:21,753 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:46:22,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:23,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:29,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:32,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:34,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:36,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:46:36,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:46:36,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:46:40,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:42,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:45,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:50,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:46:51,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:46:51,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:46:51,018 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:46:51,742 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:46:52,449 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:46:55,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:00,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:01,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:06,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:47:06,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:47:06,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:47:06,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:07,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:47:07,365 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:47:07,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:47:07,368 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:47:07,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:47:08,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:09,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:12,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:17,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:17,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:21,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:47:21,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:47:21,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:47:21,743 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:47:22,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:27,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:27,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:33,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:36,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:47:36,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:47:36,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:47:37,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:38,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:44,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:47,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:49,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:47:51,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:47:51,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:47:51,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:47:51,743 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:47:55,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:47:55,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:01,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:05,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:06,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:48:06,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:48:06,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:48:06,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:11,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:16,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:16,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:21,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:48:21,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:48:21,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:48:21,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:21,744 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:48:26,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:27,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:32,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:36,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:36,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:48:36,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:48:36,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:48:37,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:42,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:46,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:48,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:51,747 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:48:52,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:48:52,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:48:52,473 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:48:53,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:48:56,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:48:59,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:04,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:05,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:07,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:49:07,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:49:07,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:49:10,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:14,497 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:16,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:21,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:21,748 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:49:22,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:49:22,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:49:22,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:49:26,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:26,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:32,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:34,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:37,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:49:37,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:49:37,609 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:49:37,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:43,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:44,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:48,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:51,748 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:49:52,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:49:52,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:49:52,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:49:52,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:49:53,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:49:58,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:01,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:04,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:08,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:50:08,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:50:08,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:50:09,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:10,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:16,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:19,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:50:19,826 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:50:19,827 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:50:19,829 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:50:19,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:50:19,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:21,749 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:50:21,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:21,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:23,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:50:23,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:50:23,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:50:27,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:30,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:33,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:38,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:38,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:50:38,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:50:38,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:50:38,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:43,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:48,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:49,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:51,749 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:50:53,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:50:53,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:50:53,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:50:54,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:50:58,209 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:50:59,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:05,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:08,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:51:08,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:51:08,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:51:08,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:10,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:16,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:16,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:21,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:21,751 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:51:23,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:51:23,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:51:23,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:51:26,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:26,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:32,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:34,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:38,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:38,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:51:38,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:51:38,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:51:43,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:44,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:48,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:51,751 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:51:52,621 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:51:53,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:51:53,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:51:53,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:51:54,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:51:59,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:02,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:04,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:08,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:52:08,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:52:08,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:52:10,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:12,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:16,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:21,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:21,751 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:52:22,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:23,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:52:23,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:52:23,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:52:26,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:30,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:31,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:36,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:38,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:52:38,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:52:38,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:52:40,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:41,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:46,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:50,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:51,752 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:52:52,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:52:53,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:52:53,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:52:53,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:52:58,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:52:58,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:04,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:08,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:53:08,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:53:08,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:53:09,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:10,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:14,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:18,372 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:19,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:21,752 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:53:23,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:53:23,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:53:23,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:53:25,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:26,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:53:26,427 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:53:26,428 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:53:26,432 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:53:26,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:53:27,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:28,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:30,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:36,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:36,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:38,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:53:38,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:53:38,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:53:42,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:48,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:48,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:51,753 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:53:53,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:53:53,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:53:53,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:53:53,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:53:56,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:53:58,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:04,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:08,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:54:08,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:54:08,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:54:09,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:10,831 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:54:14,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:19,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:20,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:54:21,753 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:54:23,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:54:23,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:54:25,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:30,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:31,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:54:36,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:38,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:54:38,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:54:38,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:54:39,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:54:41,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:46,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:50,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:54:51,753 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:54:51,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:53,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:54:53,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:54:57,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:54:58,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:03,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:08,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:55:08,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:55:08,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:08,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:13,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:18,539 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:19,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:21,754 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:55:24,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:55:24,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:55:24,708 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:55:24,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:26,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:30,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:36,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:36,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:39,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:55:39,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:55:39,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:55:41,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:46,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:48,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:51,755 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:55:52,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:55:54,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:55:54,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:55:54,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:55:56,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:55:58,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:03,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:06,905 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:08,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:09,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:56:09,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:56:09,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:56:14,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:16,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:19,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:21,755 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:56:24,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:24,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:56:24,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:56:24,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:56:27,051 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:30,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:36,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:37,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:39,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:56:39,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:56:39,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:56:42,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:43,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 16:56:43,439 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 16:56:43,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 16:56:43,442 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 16:56:44,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 16:56:45,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:46,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:47,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:51,757 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:56:52,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:56:54,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:56:54,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:56:54,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:56:55,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:56:58,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:03,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:05,277 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:08,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:09,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:57:09,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:57:09,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:57:14,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:15,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:19,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:21,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:57:24,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:24,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:24,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:57:24,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:57:24,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:57:30,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:32,471 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:35,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:39,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:57:39,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:57:39,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:57:40,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:42,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:45,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:50,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:57:51,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:57:51,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:57:54,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:57:54,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:57:54,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:57:57,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:02,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:02,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:07,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:09,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:58:09,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:58:10,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:13,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:18,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:20,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:21,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:58:23,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:24,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:58:24,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:58:24,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:58:28,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:29,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:34,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:38,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:39,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:58:39,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:58:39,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:58:40,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:45,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:49,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:58:50,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:58:51,759 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:58:54,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:58:54,992 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:58:56,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:01,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:05,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:59:06,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:09,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:59:09,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:59:12,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:17,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:59:17,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:21,759 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:59:22,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:24,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:59:24,994 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:59:25,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:59:27,404 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:59:28,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:34,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:37,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:59:39,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:39,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:59:39,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:59:45,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:45,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 16:59:50,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:51,761 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 16:59:54,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 16:59:54,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 16:59:54,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 16:59:56,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 16:59:57,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:01,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:03,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:00:03,876 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:00:03,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:00:03,878 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:00:04,646 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:00:04,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:06,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:06,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:09,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:00:09,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:00:09,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:00:12,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:14,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:17,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:21,762 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:00:22,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:23,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:24,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:00:24,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:00:24,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:00:29,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:32,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:34,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:39,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:39,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:00:40,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:00:40,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:00:42,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:45,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:50,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:00:51,762 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:00:52,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:00:54,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:00:54,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:00:54,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:00:56,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:01,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:03,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:06,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:09,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:01:09,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:01:09,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:01:11,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:12,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:17,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:21,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:21,762 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:01:22,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:24,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:01:24,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:01:24,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:01:28,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:31,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:33,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:38,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:39,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:01:39,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:01:39,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:01:41,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:44,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:49,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:51,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:01:51,763 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:01:54,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:01:54,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:01:55,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:01:55,048 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:01:59,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:00,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:05,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:09,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:09,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:02:09,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:02:09,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:02:11,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:16,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:16,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:21,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:21,766 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:02:24,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:02:24,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:02:26,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:27,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:32,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:36,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:38,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:39,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:02:39,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:02:44,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:47,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:02:50,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:51,765 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:02:54,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:02:54,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:02:55,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:02:57,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:00,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:06,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:07,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:09,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:03:09,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:03:11,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:16,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:17,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:03:17,224 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:03:17,224 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:03:17,227 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:03:18,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:03:18,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:19,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:21,766 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:03:21,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:25,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:03:25,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:03:25,308 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:03:27,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:27,286 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:32,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:37,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:38,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:40,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:03:40,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:03:40,306 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:03:43,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:45,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:03:48,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:51,768 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:03:54,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:03:55,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:03:55,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:03:55,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:03:55,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:00,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:05,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:06,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:10,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:04:10,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:04:10,306 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:04:11,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:15,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:16,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:21,768 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:04:22,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:25,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:04:25,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:04:25,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:04:25,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:28,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:33,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:33,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:35,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:37,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:39,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:41,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:04:41,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:04:41,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:04:44,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:44,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:49,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:51,769 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:04:53,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:04:54,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:04:56,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:04:56,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:05:00,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:01,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:05,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:10,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:11,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:05:11,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:05:15,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:17,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:21,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:21,772 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:05:25,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:26,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:05:26,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:05:26,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:05:26,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:31,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:35,297 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:37,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:42,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:05:42,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:42,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:05:42,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:05:45,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:47,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:51,772 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:05:52,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:05:55,447 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:05:57,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:05:57,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:05:57,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:05:58,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:03,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:05,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:08,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:06:12,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:06:12,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:06:14,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:15,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:19,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:21,773 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:06:23,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:24,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:27,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:06:27,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:06:27,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:06:30,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:31,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:06:31,720 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:06:31,720 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:06:31,723 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:06:31,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:06:32,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:33,737 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:35,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:40,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:42,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:06:42,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:06:42,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:06:43,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:46,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:51,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:51,773 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:06:53,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:06:56,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:06:57,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:06:57,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:06:57,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:07:01,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:02,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:07,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:12,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:07:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:07:12,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:07:13,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:18,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:21,774 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:07:22,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:23,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:27,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:07:27,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:07:27,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:07:28,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:30,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:33,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:39,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:39,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:42,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:07:42,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:07:42,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:07:44,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:49,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:07:50,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:51,774 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:07:55,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:07:57,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:07:57,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:07:57,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:07:57,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:01,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:06,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:07,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:11,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:13,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:08:13,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:08:13,708 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:08:16,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:17,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:21,775 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:08:22,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:27,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:27,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:28,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:08:28,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:08:28,700 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:08:32,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:38,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:41,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:43,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:08:43,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:08:43,700 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:08:43,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:48,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:51,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:08:51,775 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:08:54,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:08:59,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:08:59,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:08:59,317 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:09:00,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:01,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:05,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:10,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:11,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:14,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:09:14,265 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:09:16,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:21,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:21,776 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:09:21,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:26,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:30,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:09:30,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:09:31,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:34,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:37,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:42,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:44,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:45,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:09:45,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:09:47,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:51,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:09:51,549 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:09:51,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:09:51,553 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:09:51,776 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:09:52,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:09:53,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:53,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:09:54,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:09:58,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:00,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:10:00,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:10:00,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:10:04,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:10:04,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:09,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:14,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:15,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:10:15,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:10:19,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:21,777 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:10:25,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:25,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:10:30,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:31,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:10:31,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:10:31,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:10:35,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:10:36,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:41,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:45,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:10:46,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:46,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:10:46,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:10:46,970 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:10:51,778 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:10:51,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:57,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:10:57,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:02,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:03,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:11:03,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:11:03,223 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:11:05,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:07,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:12,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:15,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:18,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:18,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:11:18,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:11:21,779 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:11:23,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:23,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:28,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:33,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:11:33,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:11:33,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:34,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:39,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:43,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:45,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:49,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:11:49,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:11:49,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:11:50,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:11:51,780 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:11:52,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:11:55,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:01,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:02,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:04,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:12:04,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:12:04,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:12:06,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:10,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:12,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:18,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:19,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:12:19,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:12:19,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:12:20,239 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:21,780 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:12:23,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:28,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:30,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:33,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:34,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:12:34,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:12:34,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:12:39,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:40,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:44,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:49,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:12:49,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:12:49,548 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:12:49,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:50,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:12:51,781 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:12:55,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:12:58,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:01,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:05,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:13:05,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:13:05,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:13:06,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:08,605 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:11,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:16,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:17,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:13:17,216 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:13:17,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:13:17,219 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:13:17,711 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:13:17,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:19,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:20,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:13:20,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:13:21,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:21,781 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:13:27,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:27,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:32,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:35,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:13:35,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:13:35,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:13:35,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:37,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:42,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:45,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:13:48,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:50,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:13:50,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:13:50,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:13:51,782 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:13:53,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:13:58,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:01,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:03,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:05,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:14:05,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:14:05,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:14:09,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:12,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:14,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:19,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:20,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:14:20,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:14:20,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:14:21,782 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:14:22,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:25,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:30,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:32,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:35,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:14:35,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:14:36,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:40,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:41,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:46,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:50,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:14:50,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:14:50,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:14:51,784 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:14:52,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:14:56,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:14:58,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:03,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:05,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:15:05,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:15:06,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:08,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:13,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:14,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:18,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:20,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:15:20,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:15:20,858 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:15:21,784 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:15:24,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:24,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:29,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:34,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:34,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:35,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:15:35,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:15:35,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:15:40,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:44,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:46,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:50,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:15:50,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:15:50,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:15:51,785 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:15:51,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:15:52,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:15:57,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:00,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:03,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:06,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:16:06,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:16:06,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:16:08,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:09,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:13,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:19,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:19,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:21,785 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:16:21,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:16:21,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:16:21,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:16:24,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:28,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:29,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:34,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:35,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:16:35,910 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:16:35,910 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:16:35,913 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:16:36,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:16:36,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:16:36,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:16:36,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:16:37,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:38,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:40,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:46,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:46,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:51,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:16:51,786 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:16:52,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:16:52,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:16:52,851 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:16:56,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:16:56,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:01,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:04,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:07,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:07,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:17:07,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:17:07,850 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:17:12,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:14,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:17,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:21,787 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:17:22,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:22,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:17:22,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:17:22,858 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:17:24,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:28,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:32,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:33,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:38,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:17:38,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:17:38,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:17:38,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:40,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:44,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:50,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:17:50,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:17:51,788 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:17:53,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:17:53,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:17:53,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:17:55,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:00,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:01,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:06,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:08,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:18:08,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:18:08,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:18:10,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:11,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:16,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:20,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:21,788 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:18:21,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:23,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:18:23,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:18:23,494 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:18:27,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:28,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:32,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:37,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:38,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:18:38,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:18:38,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:18:38,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:43,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:46,990 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:18:49,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:51,789 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:18:53,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:18:53,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:18:53,495 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:18:54,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:18:59,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:00,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:05,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:06,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:08,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:19:08,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:19:08,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:19:11,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:16,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:16,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:21,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:21,791 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:19:23,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:19:23,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:19:23,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:19:24,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:27,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:32,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:34,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:37,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:38,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:19:38,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:19:38,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:19:41,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:19:41,757 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:19:41,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:19:41,761 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:19:42,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:19:42,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:42,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:43,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:47,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:51,805 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:19:53,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:19:53,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:19:53,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:19:53,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:19:54,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:19:58,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:02,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:04,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:08,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:20:08,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:20:08,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:20:09,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:12,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:15,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:20,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:21,793 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:20:22,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:23,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:20:23,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:20:23,494 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:20:25,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:30,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:32,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:35,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:38,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:20:38,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:20:38,494 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:20:41,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:44,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:46,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:51,794 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:20:52,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:20:52,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:20:53,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:20:53,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:20:53,494 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:20:58,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:03,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:03,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:08,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:09,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:21:09,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:21:09,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:21:13,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:14,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:19,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:21,794 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:21:23,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:24,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:21:24,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:21:24,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:21:25,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:31,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:31,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:36,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:21:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:21:39,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:21:41,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:43,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:46,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:50,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:21:51,794 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:21:51,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:21:54,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:21:54,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:21:57,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:00,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:03,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:09,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:09,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:22:09,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:22:10,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:14,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:18,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:20,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:21,795 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:22:24,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:22:24,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:22:24,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:22:25,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:30,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:30,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:35,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:22:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:22:39,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:22:40,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:40,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:45,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:50,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:22:51,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:51,796 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:22:54,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:22:54,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:22:56,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:22:58,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:22:58,166 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:22:58,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:22:58,169 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:22:58,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:22:59,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:00,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:02,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:07,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:09,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:23:09,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:23:12,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:13,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:18,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:21,799 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:23:23,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:24,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:23:24,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:23:25,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:28,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:33,075 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:34,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:39,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:39,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:23:39,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:23:39,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:23:43,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:44,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:50,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:23:51,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:23:51,798 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:23:54,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:23:54,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:23:54,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:23:55,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:00,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:01,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:05,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:09,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:09,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:24:09,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:24:09,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:24:11,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:17,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:21,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:21,798 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:24:23,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:24,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:24:24,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:24:24,643 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:24:28,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:29,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:33,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:38,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:39,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:39,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:24:39,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:24:39,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:24:44,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:46,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:24:49,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:51,799 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:24:54,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:24:54,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:24:54,640 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:24:55,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:24:56,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:00,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:04,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:06,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:09,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:25:09,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:25:09,640 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:25:11,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:16,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:17,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:21,801 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:25:22,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:24,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:25:24,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:25:24,640 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:25:24,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:27,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:33,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:34,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:38,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:40,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:25:40,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:25:40,858 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:25:43,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:44,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:49,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:51,801 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:25:53,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:25:54,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:25:55,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:25:55,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:26:00,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:03,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:05,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:10,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:10,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:26:10,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:26:11,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:26:11,302 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:26:11,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:26:11,307 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:26:12,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:26:12,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:13,236 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:16,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:21,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:21,801 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:26:23,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:25,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:26:25,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:26:27,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:31,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:32,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:37,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:40,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:26:40,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:26:41,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:43,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:48,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:49,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:26:51,802 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:26:54,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:26:56,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:26:56,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:26:56,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:26:59,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:00,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:05,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:09,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:11,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:11,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:27:11,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:27:11,446 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:27:16,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:19,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:21,802 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:27:21,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:26,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:27:26,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:27:26,449 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:27:27,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:28,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:33,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:38,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:39,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:41,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:27:41,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:27:41,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:27:44,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:49,010 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:27:49,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:51,803 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:27:54,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:27:56,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:27:56,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:27:56,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:27:57,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:00,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:05,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:09,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:11,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:11,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:28:11,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:28:11,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:28:16,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:19,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:21,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:21,803 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:28:26,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:28:26,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:28:26,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:28:27,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:27,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:32,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:37,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:38,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:41,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:28:41,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:28:41,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:28:43,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:45,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:49,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:51,806 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:28:55,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:28:55,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:28:56,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:28:56,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:28:56,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:29:00,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:05,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:05,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:10,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:12,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:29:12,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:29:12,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:29:15,621 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:16,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:21,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:21,807 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:29:22,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:29:22,346 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:29:22,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:29:22,348 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:29:22,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:29:23,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:24,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:27,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:29:27,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:29:27,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:29:27,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:33,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:33,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:38,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:41,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:42,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:29:42,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:29:42,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:29:43,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:48,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:51,807 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:29:51,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:29:53,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:29:57,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:29:57,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:29:57,913 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:29:59,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:00,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:04,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:09,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:11,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:12,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:30:12,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:30:12,912 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:30:15,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:20,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:21,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:21,835 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:30:26,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:27,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:30:27,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:30:27,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:30:31,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:32,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:37,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:39,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:42,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:30:42,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:30:42,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:30:43,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:48,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:49,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:30:51,823 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:30:53,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:57,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:30:57,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:30:57,913 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:30:59,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:30:59,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:05,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:09,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:10,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:12,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:31:12,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:31:12,913 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:31:16,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:19,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:21,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:21,824 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:31:26,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:27,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:27,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:31:27,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:31:27,913 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:31:32,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:37,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:37,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:42,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:42,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:31:42,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:31:42,913 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:31:47,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:48,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:51,824 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:31:53,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:31:57,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:31:57,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:31:57,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:31:57,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:31:59,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:05,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:07,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:32:10,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:12,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:32:12,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:32:12,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:32:15,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:15,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:32:21,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:21,827 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:32:24,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:32:26,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:27,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:32:27,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:32:27,918 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:32:32,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:37,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:42,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:42,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:32:42,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:32:42,967 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:32:48,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:51,827 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:32:53,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:32:56,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:32:56,436 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:32:56,436 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:32:56,439 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:32:57,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:32:57,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:32:57,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:32:57,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:32:57,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:32:58,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:32:59,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:05,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:07,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:10,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:12,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:33:12,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:33:12,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:33:15,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:17,372 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:21,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:21,827 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:33:25,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:26,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:27,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:33:27,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:33:27,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:33:32,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:35,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:37,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:42,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:42,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:33:42,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:33:42,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:33:45,605 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:47,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:51,830 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:33:52,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:33:55,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:33:57,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:33:57,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:33:57,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:33:58,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:03,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:05,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:08,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:12,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:34:12,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:34:12,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:34:14,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:15,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:20,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:21,831 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:34:25,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:25,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:27,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:34:27,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:34:27,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:34:31,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:34,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:37,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:42,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:42,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:34:42,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:34:42,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:34:44,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:47,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:51,832 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:34:52,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:34:56,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:34:57,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:34:57,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:34:57,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:34:58,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:03,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:06,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:08,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:12,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:35:12,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:35:12,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:35:14,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:14,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:20,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:21,833 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:35:24,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:25,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:27,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:35:27,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:35:27,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:35:31,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:34,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:37,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:42,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:42,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:35:42,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:35:42,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:35:43,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:47,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:51,835 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:35:52,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:35:55,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:35:57,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:35:57,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:35:57,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:35:58,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:03,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:05,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:08,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:12,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:36:12,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:36:12,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:36:13,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:13,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:36:13,955 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:36:13,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:36:13,959 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:36:14,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:36:14,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:15,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:19,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:21,837 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:36:24,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:25,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:27,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:36:27,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:36:27,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:36:30,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:34,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:35,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:40,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:43,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:36:43,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:36:43,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:36:45,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:46,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:51,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:51,838 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:36:56,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:36:56,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:36:58,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:36:58,515 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:37:01,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:06,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:37:07,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:12,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:13,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:37:13,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:37:13,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:37:16,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:37:17,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:21,839 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:37:22,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:24,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:37:28,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:28,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:37:28,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:37:33,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:36,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:37:39,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:43,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:37:43,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:37:43,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:37:44,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:46,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:37:49,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:51,839 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:37:55,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:37:58,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:37:58,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:37:58,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:38:01,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:02,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:38:06,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:12,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:12,679 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:38:13,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:38:13,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:38:17,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:21,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:38:21,840 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:38:22,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:27,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:28,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:38:28,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:38:33,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:38,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:41,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:38:43,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:38:43,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:38:43,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:38:44,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:49,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:51,841 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:38:51,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:38:54,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:38:59,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:38:59,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:38:59,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:38:59,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:02,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:04,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:09,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:12,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:14,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:39:14,688 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:39:15,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:20,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:21,841 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:39:22,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:26,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:31,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:32,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:39:32,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:39:32,151 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:39:34,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:36,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:41,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:46,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:47,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:39:47,150 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:39:47,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:51,843 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:39:52,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:39:53,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:39:53,856 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:39:53,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:39:53,859 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:39:54,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:39:54,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:56,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:39:57,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:02,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:40:02,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:40:02,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:40:03,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:05,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:08,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:14,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:16,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:17,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:40:17,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:40:19,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:21,843 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:40:25,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:26,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:30,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:32,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:40:32,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:40:32,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:40:36,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:36,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:41,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:46,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:46,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:47,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:40:47,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:40:51,845 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:40:52,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:40:56,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:40:58,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:03,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:41:03,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:41:03,246 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:41:04,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:06,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:09,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:15,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:16,462 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:18,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:41:18,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:41:18,245 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:41:20,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:21,845 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:41:26,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:26,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:31,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:34,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:41:34,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:41:34,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:41:36,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:37,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:42,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:46,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:47,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:49,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:41:49,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:41:49,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:41:51,846 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:41:53,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:41:56,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:41:59,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:04,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:42:04,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:42:04,254 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:42:04,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:06,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:09,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:14,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:15,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:19,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:42:19,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:42:19,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:42:20,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:21,848 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:42:26,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:26,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:31,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:34,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:42:34,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:42:34,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:42:36,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:36,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:41,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:46,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:47,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:49,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:42:49,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:42:51,849 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:42:52,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:42:56,352 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:42:58,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:04,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:43:04,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:43:04,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:04,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:09,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:14,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:18,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:43:18,348 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:43:18,349 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:43:18,350 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:43:18,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:43:19,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:43:19,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:43:19,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:43:19,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:20,528 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:20,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:21,849 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:43:26,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:28,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:32,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:34,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:43:34,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:43:34,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:43:38,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:38,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:43,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:48,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:43:48,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:49,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:43:49,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:43:49,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:43:51,849 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:43:54,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:43:59,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:00,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:05,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:44:05,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:44:05,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:44:06,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:09,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:11,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:16,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:19,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:20,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:44:20,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:44:20,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:44:21,850 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:44:21,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:26,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:28,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:32,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:35,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:44:35,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:44:35,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:44:37,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:38,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:42,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:48,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:48,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:50,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:44:50,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:44:50,571 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:44:51,850 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:44:53,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:44:56,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:44:59,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:04,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:05,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:45:05,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:45:05,574 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:45:08,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:09,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:14,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:16,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:19,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:20,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:45:20,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:45:20,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:45:21,851 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:45:25,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:26,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:30,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:35,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:35,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:45:35,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:45:35,573 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:45:36,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:41,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:46,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:46,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:50,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:45:50,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:45:50,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:45:51,852 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:45:51,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:45:56,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:45:57,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:02,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:05,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:46:05,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:46:05,709 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:46:07,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:08,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:46:13,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:18,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:19,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:46:20,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:46:20,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:46:20,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:46:21,852 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:46:23,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:29,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:46:29,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:34,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:35,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:46:35,463 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:46:35,464 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:46:35,467 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:46:35,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:46:35,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:46:35,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:46:36,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:46:37,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:46:39,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:44,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:47,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:46:49,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:50,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:46:50,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:46:50,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:46:51,852 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:46:54,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:46:59,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:00,404 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:05,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:05,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:47:05,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:47:05,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:47:10,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:11,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:16,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:20,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:20,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:47:20,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:47:20,671 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:47:21,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:21,853 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:47:26,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:30,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:32,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:35,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:47:35,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:47:35,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:47:38,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:42,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:44,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:49,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:47:50,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:47:50,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:47:50,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:47:50,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:47:51,853 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:47:54,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:00,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:00,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:05,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:48:05,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:48:05,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:48:05,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:10,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:11,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:16,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:20,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:48:20,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:48:20,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:48:20,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:21,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:21,854 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:48:26,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:31,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:32,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:35,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:48:35,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:48:35,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:48:37,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:42,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:43,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:47,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:50,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:48:50,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:48:51,854 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:48:52,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:48:53,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:48:57,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:03,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:03,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:05,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:49:05,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:49:09,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:11,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:15,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:20,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:20,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:49:20,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:49:20,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:21,855 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:49:25,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:30,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:32,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:35,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:49:35,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:49:35,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:41,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:42,744 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:46,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:50,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:49:50,677 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:49:51,855 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:49:51,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:49:52,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:49:56,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:00,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:50:00,517 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:50:00,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:50:00,518 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:50:00,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:50:01,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:02,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:02,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:05,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:50:05,677 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:50:08,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:10,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:14,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:19,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:20,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:50:20,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:50:21,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:21,856 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:50:24,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:29,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:35,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:35,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:50:35,677 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:50:35,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:41,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:45,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:51,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:50:51,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:50:51,244 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:50:51,857 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:50:51,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:50:53,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:50:57,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:02,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:03,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:06,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:51:06,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:51:06,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:51:08,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:13,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:14,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:19,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:21,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:51:21,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:51:21,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:51:21,858 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:51:23,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:24,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:29,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:35,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:35,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:36,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:51:36,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:51:36,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:51:40,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:45,733 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:46,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:51,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:51:51,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:51:51,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:51:51,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:51:51,936 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:51:54,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:51:56,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:01,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:02,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:06,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:52:06,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:52:07,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:12,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:13,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:18,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:21,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:52:21,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:52:21,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:52:23,010 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:23,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:28,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:33,080 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:34,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:36,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:52:36,677 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:52:39,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:45,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:45,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:50,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:52:51,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:52:51,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:52:51,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:52:51,892 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:52:55,235 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:52:56,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:01,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:06,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:06,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:53:06,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:53:06,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:53:09,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:53:11,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:16,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:19,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:53:21,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:53:21,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:53:21,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:53:21,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:26,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:27,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:53:27,265 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:53:27,266 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:53:27,268 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:53:27,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:53:29,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:53:32,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:36,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:53:36,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:53:36,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:53:37,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:39,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:53:42,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:48,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:49,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:53:51,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:53:51,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:53:51,873 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:53:53,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:59,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:53:59,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:04,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:06,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:54:06,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:54:06,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:54:09,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:09,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:14,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:17,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:20,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:21,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:54:21,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:54:21,886 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:54:25,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:28,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:31,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:36,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:36,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:54:36,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:54:39,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:41,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:46,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:49,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:54:51,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:51,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:54:51,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:54:51,928 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:54:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:54:59,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:02,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:06,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:55:06,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:55:08,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:09,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:13,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:18,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:19,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:21,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:55:21,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:55:21,875 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:55:23,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:29,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:29,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:34,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:36,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:55:36,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:55:36,684 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:55:39,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:39,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:44,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:49,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:55:50,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:51,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:55:51,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:55:51,867 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:55:55,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:55:59,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:01,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:06,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:06,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:56:06,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:56:07,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:11,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:16,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:19,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:21,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:56:21,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:56:21,938 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:56:21,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:26,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:29,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:32,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:36,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:56:36,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:56:38,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:39,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:43,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:47,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 17:56:47,580 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 17:56:47,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 17:56:47,582 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 17:56:47,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 17:56:49,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:56:49,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:56:51,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:56:51,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:56:51,865 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:56:54,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:00,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:00,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:05,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:06,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:57:06,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:57:08,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:10,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:15,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:20,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:21,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:21,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:57:21,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:57:21,866 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:57:26,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:27,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:31,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:36,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:57:36,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:57:36,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:37,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:41,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:43,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:44,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:46,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:51,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:51,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:57:51,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:57:51,890 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:57:52,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:57:52,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:57:57,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:01,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:03,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:08,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:58:08,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:58:08,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:58:09,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:11,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:14,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:19,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:19,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:21,867 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:58:23,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:58:23,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:58:23,175 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:58:25,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:29,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:31,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:36,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:39,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:58:39,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:58:39,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:58:41,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:41,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:46,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:51,867 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:58:51,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:58:53,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:58:54,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:58:54,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:58:54,449 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:58:57,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:03,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:03,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:08,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:09,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:59:09,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:59:09,449 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:59:13,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:16,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:18,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:21,869 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:59:23,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:24,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:59:24,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:59:24,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:59:26,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:29,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:35,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:36,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:39,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:59:39,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:59:39,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:59:40,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:46,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:46,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:51,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 17:59:51,870 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 17:59:54,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 17:59:54,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 17:59:54,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 17:59:56,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 17:59:56,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:01,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:04,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:00:04,460 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:00:04,460 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:00:04,463 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:00:05,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:00:06,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:00:07,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:09,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:00:09,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:00:09,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:00:12,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:15,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:00:17,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:21,873 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:00:23,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:24,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:00:24,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:00:24,453 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:00:27,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:00:29,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:34,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:37,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:00:39,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:00:39,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:39,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:00:39,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:00:44,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:47,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:00:49,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:51,872 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:00:54,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:00:54,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:00:54,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:00:54,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:00:57,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:00,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:05,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:07,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:09,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:01:09,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:01:11,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:16,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:17,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:21,884 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:01:21,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:26,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:01:26,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:01:26,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:01:27,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:30,030 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:32,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:37,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:40,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:41,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:01:41,140 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:01:43,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:48,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:50,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:51,873 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:01:53,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:01:56,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:01:56,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:01:56,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:01:58,235 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:01:59,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:04,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:09,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:10,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:11,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:02:11,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:02:15,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:18,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:21,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:21,874 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:02:26,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:02:26,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:02:26,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:02:26,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:28,470 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:32,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:37,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:38,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:41,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:02:41,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:02:43,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:48,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:48,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:51,874 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:02:53,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:02:56,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:02:56,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:02:57,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:02:59,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:04,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:07,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:09,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:11,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:03:11,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:03:15,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:15,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:20,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:21,874 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:03:25,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:25,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:03:25,687 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:03:25,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:03:25,690 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:03:25,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:03:26,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:03:26,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:03:26,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:27,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:31,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:37,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:37,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:41,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:03:41,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:03:42,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:47,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:48,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:51,875 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:03:52,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:03:56,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:03:56,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:03:58,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:03:58,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:03,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:08,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:09,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:11,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:04:11,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:04:15,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:18,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:21,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:21,876 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:04:26,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:04:26,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:04:26,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:28,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:31,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:36,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:36,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:41,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:04:41,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:04:42,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:46,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:48,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:51,878 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:04:53,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:04:55,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:04:56,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:04:56,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:04:59,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:04,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:05,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:09,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:11,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:05:11,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:05:15,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:16,016 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:20,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:21,879 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:05:26,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:26,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:05:26,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:05:26,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:31,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:36,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:37,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:41,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:05:41,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:05:42,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:46,248 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:48,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:51,880 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:05:53,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:05:56,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:05:56,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:05:56,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:05:59,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:04,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:09,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:11,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:06:11,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:06:15,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:18,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:06:20,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:21,881 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:06:25,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:26,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:06:26,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:06:28,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:06:31,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:36,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:40,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:06:41,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:06:41,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:06:42,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:47,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:50,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:06:51,881 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:06:52,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:57,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:06:57,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:06:57,028 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:06:58,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:06:58,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:06:58,206 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:06:58,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:06:58,209 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:06:58,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:07:00,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:03,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:08,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:11,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:12,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:07:12,025 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:07:14,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:19,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:20,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:21,882 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:07:25,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:27,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:07:27,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:07:30,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:31,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:36,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:40,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:41,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:42,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:07:42,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:07:47,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:50,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:07:51,882 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:07:52,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:07:57,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:07:57,025 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:07:58,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:03,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:04,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:08,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:12,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:08:12,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:08:14,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:14,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:19,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:21,894 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:08:24,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:25,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:27,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:08:27,025 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:08:30,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:34,666 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:36,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:41,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:42,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:08:42,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:08:42,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:46,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:51,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:51,884 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:08:52,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:08:56,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:08:57,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:08:57,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:09:02,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:02,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:07,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:12,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:09:12,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:09:12,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:12,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:17,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:21,884 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:09:22,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:23,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:27,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:09:27,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:09:28,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:33,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:33,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:38,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:42,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:09:42,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:09:43,189 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:44,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:49,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:51,885 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:09:53,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:09:54,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:09:57,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:09:57,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:10:00,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:02,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:05,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:10,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:12,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:10:12,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:10:12,449 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:16,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:20,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:10:20,797 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:10:20,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:10:20,799 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:10:21,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:10:21,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:21,885 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:10:22,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:23,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:26,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:27,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:10:27,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:10:32,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:32,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:37,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:42,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:10:42,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:10:42,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:43,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:48,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:51,886 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:10:52,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:10:53,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:10:57,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:10:57,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:10:59,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:02,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:04,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:09,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:12,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:11:12,028 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:11:12,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:15,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:21,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:21,886 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:11:22,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:26,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:27,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:11:27,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:11:27,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:11:31,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:33,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:36,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:41,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:42,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:42,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:11:42,046 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:11:47,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:51,189 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:11:51,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:11:52,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:11:57,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:11:57,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:11:57,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:11:58,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:01,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:03,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:09,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:11,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:12,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:12:12,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:12:12,862 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:12:15,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:20,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:21,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:21,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:12:25,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:27,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:12:27,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:12:27,862 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:12:30,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:31,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:36,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:40,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:41,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:42,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:12:42,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:12:42,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:12:47,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:50,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:12:51,899 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:12:52,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:57,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:12:57,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:12:57,862 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:12:58,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:12:58,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:03,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:08,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:08,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:12,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:13:12,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:13:12,862 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:13:14,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:18,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:19,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:21,907 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:13:24,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:27,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:13:27,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:13:27,862 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:13:29,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:30,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:35,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:37,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:13:37,131 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:13:37,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:13:37,134 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:13:37,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:13:39,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:40,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:42,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:13:42,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:13:42,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:13:46,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:49,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:13:51,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:51,907 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:13:56,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:13:57,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:13:57,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:13:57,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:13:59,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:02,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:07,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:09,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:12,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:12,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:14:12,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:14:12,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:14:18,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:19,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:21,908 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:14:23,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:27,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:14:27,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:14:27,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:14:29,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:29,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:34,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:38,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:39,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:42,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:14:42,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:14:42,867 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:14:45,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:46,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:51,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:51,909 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:14:56,843 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:14:56,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:14:57,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:14:57,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:14:57,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:15:02,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:06,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:07,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:12,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:14,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:15:14,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:15:14,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:15:16,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:18,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:21,909 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:15:23,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:27,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:29,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:15:29,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:15:29,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:15:29,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:34,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:37,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:40,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:44,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:15:44,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:15:44,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:15:45,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:47,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:51,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:51,910 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:15:56,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:15:57,278 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:15:59,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:15:59,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:15:59,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:16:02,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:07,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:07,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:12,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:14,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:16:14,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:16:14,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:16:17,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:17,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:21,910 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:16:23,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:27,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:29,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:29,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:16:29,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:16:34,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:37,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:39,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:44,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:16:44,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:16:45,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:47,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:50,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:51,911 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:16:55,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:16:55,574 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:16:55,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:16:55,577 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:16:55,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:16:56,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:16:56,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:57,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:16:59,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:16:59,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:17:02,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:06,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:07,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:12,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:14,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:17:14,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:17:17,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:18,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:21,912 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:17:23,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:25,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:29,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:29,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:17:29,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:17:29,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:17:35,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:35,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:40,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:44,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:17:44,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:17:44,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:17:45,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:45,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:50,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:51,914 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:17:55,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:17:55,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:17:59,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:17:59,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:17:59,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:18:01,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:05,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:06,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:11,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:14,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:18:14,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:18:14,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:18:15,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:17,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:21,915 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:18:22,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:25,637 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:27,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:29,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:18:29,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:18:29,811 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:18:33,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:35,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:38,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:44,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:18:44,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:18:44,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:18:45,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:45,852 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:50,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:51,916 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:18:55,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:18:55,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:18:59,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:18:59,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:18:59,811 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:19:00,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:05,990 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:06,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:11,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:13,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:14,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:19:14,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:19:14,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:19:17,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:21,917 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:19:22,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:23,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:27,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:31,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:19:31,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:19:31,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:19:32,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:33,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:37,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:42,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:43,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:46,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:19:46,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:19:48,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:51,917 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:19:53,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:19:54,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:19:59,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:01,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:20:01,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:20:01,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:20:03,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:04,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:09,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:11,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:20:11,936 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:20:11,937 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:20:11,939 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:20:12,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:20:13,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:14,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:16,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:20:16,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:20:20,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:21,918 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:20:23,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:25,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:30,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:31,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:20:31,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:20:31,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:20:33,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:36,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:42,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:43,737 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:46,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:20:46,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:20:48,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:51,919 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:20:53,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:20:54,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:20:59,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:01,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:21:01,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:21:01,213 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:21:03,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:04,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:09,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:14,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:15,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:16,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:21:16,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:21:20,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:21,919 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:21:24,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:25,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:30,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:31,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:21:31,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:21:34,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:35,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:40,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:43,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:46,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:21:46,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:46,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:21:51,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:21:51,930 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:21:53,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:21:56,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:01,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:22:01,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:22:01,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:03,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:22:06,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:12,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:13,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:22:16,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:22:16,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:22:18,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:21,923 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:22:23,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:22:23,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:28,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:31,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:22:31,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:22:34,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:37,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:22:40,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:45,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:46,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:22:46,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:22:47,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:22:50,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:51,923 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:22:55,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:22:57,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:01,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:23:01,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:01,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:23:06,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:07,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:11,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:16,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:23:16,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:23:16,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:17,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:21,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:21,924 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:23:26,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:28,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:31,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:23:31,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:23:32,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:35,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:23:35,840 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:23:35,840 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:23:35,842 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:23:36,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:23:37,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:38,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:42,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:46,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:23:46,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:23:48,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:48,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:51,924 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:23:53,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:23:57,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:23:59,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:01,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:24:01,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:24:05,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:07,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:10,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:15,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:15,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:16,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:24:16,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:24:21,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:21,926 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:24:25,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:27,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:31,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:24:31,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:24:31,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:24:32,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:35,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:38,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:43,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:45,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:46,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:24:46,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:24:48,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:51,927 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:24:53,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:24:55,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:24:59,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:01,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:25:01,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:25:01,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:25:04,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:06,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:25:09,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:15,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:16,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:25:16,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:25:16,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:25:21,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:21,930 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:25:26,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:25:27,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:32,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:25:32,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:25:32,349 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:25:32,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:36,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:25:38,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:43,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:44,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:25:47,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:25:47,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:25:47,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:25:49,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:51,929 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:25:54,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:25:59,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:02,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:26:02,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:26:05,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:10,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:15,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:17,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:26:17,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:26:21,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:21,933 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:26:25,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:26:26,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:31,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:33,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:26:33,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:26:33,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:26:35,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:26:37,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:42,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:45,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:26:47,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:48,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:26:48,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:26:48,532 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:26:51,932 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:26:52,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:26:55,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:26:58,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:03,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:27:03,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:27:03,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:27:03,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:05,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:09,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:14,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:15,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:18,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:27:18,532 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:27:20,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:21,933 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:27:24,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:27:24,048 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:27:24,049 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:27:24,052 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:27:25,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:27:26,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:26,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:31,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:33,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:27:33,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:27:36,124 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:36,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:41,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:46,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:46,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:48,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:27:48,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:27:51,935 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:27:51,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:27:56,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:27:57,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:02,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:03,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:28:03,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:28:06,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:07,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:12,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:16,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:18,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:28:18,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:28:18,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:21,937 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:28:24,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:24,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:29,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:33,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:28:33,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:28:34,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:34,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:39,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:44,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:45,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:48,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:28:48,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:28:51,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:28:51,937 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:28:54,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:28:57,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:03,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:03,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:29:03,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:29:04,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:08,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:13,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:13,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:18,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:29:18,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:29:18,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:29:19,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:21,939 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:29:23,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:24,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:29,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:33,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:29:33,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:29:34,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:35,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:40,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:44,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:46,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:48,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:29:48,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:29:51,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:29:51,939 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:29:54,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:29:57,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:02,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:03,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:30:03,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:30:04,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:08,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:13,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:14,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:18,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:30:18,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:30:18,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:30:19,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:21,939 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:30:24,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:24,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:29,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:33,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:30:33,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:30:34,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:35,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:40,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:42,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:30:42,092 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:30:42,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:30:42,096 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:30:42,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:30:44,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:46,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:48,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:30:48,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:30:51,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:30:51,940 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:30:54,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:30:57,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:02,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:03,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:31:03,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:31:04,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:31:08,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:14,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:14,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:31:18,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:31:18,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:31:19,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:21,940 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:31:24,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:29,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:33,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:31:33,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:31:35,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:40,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:45,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:48,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:31:48,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:31:51,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:31:51,943 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:31:54,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:31:57,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:02,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:03,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:32:03,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:32:03,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:32:06,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:32:07,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:12,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:16,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:32:18,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:18,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:32:18,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:32:21,943 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:32:23,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:26,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:32:29,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:33,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:32:33,539 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:32:35,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:36,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:32:40,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:45,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:46,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:32:48,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:32:48,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:32:50,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:51,945 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:32:56,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:32:56,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:01,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:04,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:33:04,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:33:04,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:33:06,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:06,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:11,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:16,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:16,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:19,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:33:19,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:33:19,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:33:21,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:21,947 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:33:26,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:27,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:32,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:34,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:33:34,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:33:34,057 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:33:36,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:37,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:42,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:47,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:33:47,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:49,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:33:49,054 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:33:51,946 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:33:52,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:58,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:33:59,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:03,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:04,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:34:04,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:34:04,203 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:34:06,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:09,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:14,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:16,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:19,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:34:19,201 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:34:20,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:21,947 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:34:25,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:26,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:30,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:34,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:34:34,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:34:34,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:34:35,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:34:35,520 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:34:35,521 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:34:35,524 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:34:36,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:34:36,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:36,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:41,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:46,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:47,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:49,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:34:49,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:34:49,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:34:51,949 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:34:52,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:34:56,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:34:58,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:03,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:04,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:35:04,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:35:04,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:35:06,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:08,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:13,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:16,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:19,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:19,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:35:19,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:35:21,950 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:35:24,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:26,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:29,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:34,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:35:34,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:35:34,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:35,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:40,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:44,965 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:46,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:49,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:35:49,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:35:51,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:35:51,951 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:35:55,041 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:35:56,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:01,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:04,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:36:04,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:36:04,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:36:05,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:07,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:12,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:15,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:17,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:19,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:36:19,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:36:19,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:36:21,961 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:36:23,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:25,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:28,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:33,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:34,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:36:34,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:36:35,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:39,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:44,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:44,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:49,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:36:49,373 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:36:50,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:36:51,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:36:54,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:36:55,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:00,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:02,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:04,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:37:04,372 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:37:06,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:11,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:14,666 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:17,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:20,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:37:20,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:37:20,967 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:37:21,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:37:22,722 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:23,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:28,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:32,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:33,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:35,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:37:35,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:37:39,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:42,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:44,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:49,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:50,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:37:50,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:37:51,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:37:51,447 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:37:51,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:37:51,449 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:37:51,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:37:51,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:37:52,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:37:54,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:37:59,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:03,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:38:05,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:05,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:38:05,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:38:11,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:16,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:19,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:38:20,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:38:20,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:38:21,955 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:38:21,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:27,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:27,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:38:32,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:35,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:38:35,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:38:37,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:37,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:38:42,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:47,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:49,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:38:50,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:38:50,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:38:51,955 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:38:52,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:58,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:38:59,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:03,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:05,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:39:05,971 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:39:08,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:08,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:13,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:16,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:19,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:20,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:39:20,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:39:21,956 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:39:24,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:26,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:29,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:34,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:35,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:39:35,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:39:35,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:39:36,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:40,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:45,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:46,905 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:39:50,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:50,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:39:50,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:39:51,956 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:39:55,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:39:56,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:00,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:05,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:40:05,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:40:06,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:07,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:11,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:15,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:16,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:21,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:21,958 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:40:22,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:40:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:40:22,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:40:25,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:27,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:32,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:35,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:37,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:40:37,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:40:37,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:37,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:40:42,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:45,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:47,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:51,960 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:40:52,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:40:52,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:40:52,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:40:53,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:40:55,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:40:58,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:04,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:05,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:07,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:41:07,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:41:07,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:41:09,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:12,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:41:12,186 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:41:12,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:41:12,191 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:41:12,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:41:13,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:15,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:20,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:21,960 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:41:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:41:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:41:22,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:41:23,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:25,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:31,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:31,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:36,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:37,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:41:37,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:41:37,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:41:41,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:41,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:46,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:50,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:41:51,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:41:51,960 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:41:52,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:41:52,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:41:52,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:41:57,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:00,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:02,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:07,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:42:07,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:42:07,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:42:08,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:08,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:13,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:19,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:19,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:21,961 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:42:22,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:42:22,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:42:22,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:42:25,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:27,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:30,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:35,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:37,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:37,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:42:37,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:42:37,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:42:41,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:46,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:47,235 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:51,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:42:51,961 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:42:52,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:42:52,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:42:52,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:42:57,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:42:57,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:02,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:07,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:43:07,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:43:07,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:43:07,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:43:07,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:12,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:17,512 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:43:17,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:21,962 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:43:22,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:43:22,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:43:22,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:43:23,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:27,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:43:29,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:34,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:43:37,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:43:37,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:43:37,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:43:39,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:45,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:45,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:43:50,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:51,964 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:43:52,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:43:52,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:43:52,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:43:55,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:43:55,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:00,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:05,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:06,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:07,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:44:07,475 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:44:11,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:13,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:17,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:21,965 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:44:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:44:22,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:44:22,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:44:22,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:22,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:44:22,817 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:44:22,818 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:44:22,820 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:44:23,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:44:25,031 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:26,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:27,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:33,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:33,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:37,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:44:37,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:44:38,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:43,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:44,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:49,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:51,965 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:44:52,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:44:52,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:44:53,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:44:54,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:44:59,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:03,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:05,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:07,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:45:07,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:45:10,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:11,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:15,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:20,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:21,966 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:45:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:45:22,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:45:23,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:26,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:31,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:33,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:36,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:37,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:45:37,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:45:42,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:43,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:47,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:51,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:45:51,966 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:45:52,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:45:52,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:45:52,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:45:57,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:01,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:46:03,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:07,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:46:07,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:46:08,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:13,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:18,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:21,968 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:46:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:46:22,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:46:23,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:29,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:30,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:46:34,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:37,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:46:37,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:46:37,479 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:46:39,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:40,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:46:44,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:50,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:46:50,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:46:51,968 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:46:52,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:46:52,479 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:46:55,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:00,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:01,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:06,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:07,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:07,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:47:07,478 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:47:11,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:16,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:17,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:21,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:21,969 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:47:22,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:47:22,478 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:47:27,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:29,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:32,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:37,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:47:37,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:37,479 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:47:37,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:42,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:47,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:47,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:51,969 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:47:52,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:47:52,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:47:52,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:47:53,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:47:55,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:47:55,626 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:47:55,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:47:55,629 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:47:55,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:47:57,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:47:59,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:04,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:05,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:07,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:48:07,479 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:48:09,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:15,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:15,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:20,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:21,971 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:48:23,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:48:23,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:48:23,111 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:48:25,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:25,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:30,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:35,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:36,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:38,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:48:38,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:48:38,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:48:41,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:44,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:47,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:51,971 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:48:52,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:48:53,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:48:53,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:48:53,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:48:54,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:48:58,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:04,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:04,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:08,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:49:08,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:49:09,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:14,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:14,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:19,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:21,972 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:49:22,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:23,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:49:23,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:49:25,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:30,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:32,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:35,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:38,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:49:38,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:49:38,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:49:40,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:42,512 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:45,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:51,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:49:51,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:49:51,975 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:49:53,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:49:53,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:49:53,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:49:56,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:01,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:02,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:07,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:09,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:50:09,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:50:09,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:50:11,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:12,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:17,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:21,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:21,973 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:50:22,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:24,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:50:24,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:50:24,564 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:50:28,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:29,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:33,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:39,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:50:39,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:50:39,564 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:50:39,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:39,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:44,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:48,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:50:50,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:51,974 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:50:54,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:50:54,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:50:54,873 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:50:55,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:50:58,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:00,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:05,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:51:05,315 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:51:05,316 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:51:05,319 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:51:06,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:51:06,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:06,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:08,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:09,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:51:09,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:51:09,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:51:12,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:16,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:17,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:21,979 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:51:22,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:24,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:51:24,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:51:24,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:51:26,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:28,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:33,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:34,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:38,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:41,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:51:41,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:51:41,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:51:44,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:44,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:49,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:51,976 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:51:52,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:51:54,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:51:56,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:51:56,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:52:00,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:02,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:05,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:10,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:11,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:52:11,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:52:11,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:52:12,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:16,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:21,767 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:21,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:21,976 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:52:26,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:52:26,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:52:27,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:31,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:32,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:37,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:41,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:52:41,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:52:43,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:48,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:48,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:51,977 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:52:53,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:52:56,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:52:56,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:52:58,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:52:59,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:04,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:08,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:09,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:11,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:53:11,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:53:15,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:18,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:20,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:21,977 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:53:26,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:53:26,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:53:26,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:26,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:31,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:36,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:36,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:41,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:53:41,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:53:42,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:46,558 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:47,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:51,979 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:53:52,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:53:56,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:53:56,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:53:56,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:53:58,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:03,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:04,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:12,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:54:12,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:54:12,014 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:54:14,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:14,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:19,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:21,980 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:54:22,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:54:22,504 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:54:22,505 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:54:22,507 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:54:22,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:54:23,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:24,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:27,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:54:27,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:54:27,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:54:30,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:34,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:35,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:40,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:42,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:54:42,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:54:42,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:54:44,075 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:46,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:51,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:51,980 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:54:52,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:54:56,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:54:57,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:54:57,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:54:57,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:55:02,330 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:02,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:07,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:12,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:55:12,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:55:12,012 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:55:12,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:13,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:18,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:21,981 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:55:22,589 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:23,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:28,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:55:28,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:55:28,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:55:29,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:30,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:34,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:39,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:40,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:43,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:55:43,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:55:43,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:55:45,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:48,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:55:50,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:51,981 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:55:55,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:55:58,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:55:58,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:55:58,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:55:58,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:01,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:06,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:09,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:12,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:13,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:56:13,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:56:13,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:56:17,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:18,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:21,981 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:56:22,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:26,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:28,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:56:28,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:56:28,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:56:28,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:33,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:36,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:39,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:43,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:56:43,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:56:43,196 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:56:44,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:46,273 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:49,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:51,983 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:56:54,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:56:55,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:56:58,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:56:58,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:56:58,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:57:00,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:04,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:05,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:10,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:13,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:57:13,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:57:13,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:57:14,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:16,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:21,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:21,983 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:57:24,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:27,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:28,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:57:28,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:57:28,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:57:31,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 18:57:31,756 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 18:57:31,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 18:57:31,759 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 18:57:32,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 18:57:32,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:32,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:34,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:37,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:42,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:42,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:43,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:57:43,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:57:43,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:57:48,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:51,983 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:57:52,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:57:53,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:58,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:57:59,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:57:59,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:57:59,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:58:02,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:04,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:09,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:12,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:14,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:58:14,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:58:14,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:19,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:21,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:21,984 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:58:24,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:29,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:58:29,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:58:30,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:31,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:35,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:40,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:41,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:44,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:58:44,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:58:46,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:50,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:58:51,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:51,985 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:58:56,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:58:59,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:58:59,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:59:00,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:02,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:02,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:07,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:10,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:12,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:14,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:59:14,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:59:17,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:18,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:21,985 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:59:22,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:28,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:28,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:29,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:59:29,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:59:33,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:38,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:38,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:43,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:45,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 18:59:45,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 18:59:45,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 18:59:46,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 18:59:49,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:51,986 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 18:59:54,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 18:59:56,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:00,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:00:00,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:00,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:00:05,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:06,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:10,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:14,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:15,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:00:15,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:00:16,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:21,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:21,987 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:00:24,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:26,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:31,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:00:31,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:00:31,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:00:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:35,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:37,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:41,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:00:41,949 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:00:41,950 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:00:41,952 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:00:42,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:00:42,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:43,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:44,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:46,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:00:46,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:00:46,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:00:48,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:51,987 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:00:53,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:00:53,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:00:58,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:01,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:01,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:01:01,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:01:01,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:01:04,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:10,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:11,275 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:15,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:17,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:01:17,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:01:17,154 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:01:19,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:20,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:21,988 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:01:25,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:29,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:30,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:32,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:01:32,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:01:32,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:01:36,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:39,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:42,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:47,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:01:47,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:47,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:01:47,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:01:48,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:51,988 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:01:52,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:01:56,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:01:58,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:02,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:02:02,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:02:02,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:02:04,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:09,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:14,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:16,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:02:17,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:02:17,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:02:17,154 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:02:19,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:21,990 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:02:24,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:26,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:02:29,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:32,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:02:32,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:02:32,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:02:34,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:02:35,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:40,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:44,926 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:02:45,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:47,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:02:47,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:02:47,449 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:02:50,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:02:51,991 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:02:55,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:02:56,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:01,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:02,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:03:02,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:03:05,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:06,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:11,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:15,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:16,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:17,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:03:17,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:03:21,993 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:03:21,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:25,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:27,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:32,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:03:32,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:03:33,409 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:33,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:38,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:43,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:44,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:47,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:03:47,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:03:49,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:51,993 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:03:53,551 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:03:54,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:03:59,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:00,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:04:00,488 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:04:00,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:04:00,491 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:04:00,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:04:01,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:02,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:04:02,449 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:04:02,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:05,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:10,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:11,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:16,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:18,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:04:18,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:04:18,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:04:20,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:21,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:21,993 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:04:26,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:30,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:32,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:33,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:04:33,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:04:33,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:04:37,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:40,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:42,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:47,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:48,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:04:48,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:04:48,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:04:51,994 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:04:52,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:58,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:04:59,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:03,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:03,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:05:03,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:05:08,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:09,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:13,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:18,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:18,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:05:18,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:05:19,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:21,994 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:05:23,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:29,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:29,278 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:33,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:05:33,404 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:05:34,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:37,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:39,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:44,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:47,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:05:48,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:05:48,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:05:49,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:51,996 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:05:54,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:05:57,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:00,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:03,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:06:03,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:06:03,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:06:05,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:05,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:10,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:15,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:15,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:18,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:06:18,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:06:18,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:06:20,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:21,997 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:06:25,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:26,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:31,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:33,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:06:33,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:06:33,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:06:35,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:36,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:41,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:45,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:47,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:48,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:06:48,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:06:48,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:06:51,999 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:06:53,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:06:55,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:06:58,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:03,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:03,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:07:03,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:07:03,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:07:04,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:07:08,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:13,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:17,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:07:17,786 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:07:17,787 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:07:17,789 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:07:18,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:07:18,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:07:18,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:07:18,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:07:18,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:19,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:07:21,055 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:07:21,998 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:07:23,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:29,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:33,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:07:33,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:07:33,513 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:07:34,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:39,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:44,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:48,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:07:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:07:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:07:49,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:07:52,001 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:07:55,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:00,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:03,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:08:03,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:08:03,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:08:05,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:10,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:15,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:18,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:08:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:08:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:08:21,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:21,999 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:08:27,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:32,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:33,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:08:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:08:33,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:08:37,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:42,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:47,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:48,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:08:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:08:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:08:52,000 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:08:53,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:08:58,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:03,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:03,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:09:03,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:09:03,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:09:08,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:13,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:18,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:09:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:09:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:09:19,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:22,001 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:09:25,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:30,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:33,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:09:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:09:33,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:09:35,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:40,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:45,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:48,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:09:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:09:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:09:50,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:09:52,001 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:09:56,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:01,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:03,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:10:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:10:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:10:06,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:11,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:16,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:18,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:10:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:10:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:10:21,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:22,001 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:10:27,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:32,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:33,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:10:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:10:33,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:10:37,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:42,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:47,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:48,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:10:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:10:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:10:52,002 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:10:53,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:10:58,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:03,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:03,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:11:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:11:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:11:08,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:13,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:18,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:11:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:11:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:11:18,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:22,002 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:11:24,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:29,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:33,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:11:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:11:33,557 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:11:34,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:39,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:44,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:48,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:11:48,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:11:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:11:49,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:11:52,013 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:11:55,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:00,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:03,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:12:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:12:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:12:05,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:10,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:15,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:18,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:12:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:12:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:12:21,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:22,003 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:12:27,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:32,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:33,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:12:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:12:33,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:12:37,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:42,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:47,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:48,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:12:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:12:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:12:52,003 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:12:53,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:12:58,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:03,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:03,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:13:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:13:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:13:08,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:13,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:18,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:13:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:13:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:13:19,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:22,004 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:13:25,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:30,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:33,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:13:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:13:33,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:13:35,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:40,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:45,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:48,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:13:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:13:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:13:50,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:13:52,004 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:13:56,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:01,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:03,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:14:03,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:14:03,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:14:06,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:11,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:16,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:18,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:14:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:14:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:14:21,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:22,007 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:14:27,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:32,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:33,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:14:33,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:14:33,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:14:37,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:42,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:47,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:48,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:14:48,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:14:48,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:14:52,007 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:14:53,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:14:58,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:03,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:03,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:15:03,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:15:03,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:15:08,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:13,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:18,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:15:18,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:15:18,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:15:19,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:22,008 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:15:25,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:30,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:33,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:15:33,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:15:33,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:15:35,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:40,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:45,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:48,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:15:48,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:15:48,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:15:51,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:15:52,008 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:15:57,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:02,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:03,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:16:03,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:16:03,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:16:07,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:12,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:17,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:18,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:16:18,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:16:18,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:16:22,008 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:16:23,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:28,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:33,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:33,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:16:33,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:16:33,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:16:38,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:43,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:48,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:16:48,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:16:48,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:16:48,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:52,009 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:16:54,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:16:59,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:03,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:17:03,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:17:03,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:17:04,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:09,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:14,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:18,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:17:18,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:17:18,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:17:19,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:22,009 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:17:25,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:30,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:33,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:17:33,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:17:33,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:17:35,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:40,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:45,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:48,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:17:48,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:17:48,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:17:50,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:52,010 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:17:56,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:17:59,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:17:59,125 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:17:59,125 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:17:59,128 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:18:00,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:18:00,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:18:01,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:02,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:18:03,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:18:03,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:18:03,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:18:06,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:11,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:16,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:18,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:18:18,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:18:18,518 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:18:22,010 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:18:22,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:27,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:30,200 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:18:32,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:33,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:18:33,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:18:33,691 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:18:37,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:42,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:47,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:49,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:18:49,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:18:49,014 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:18:50,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:18:51,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:18:52,011 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:18:53,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:58,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:18:58,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:18:59,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:19:03,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:08,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:13,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:19:13,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:18,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:20,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:19:20,159 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:19:20,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:19:20,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:19:21,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:19:22,011 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:19:24,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:28,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:19:29,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:34,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:35,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:19:35,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:19:40,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:43,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:19:45,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:50,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:50,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:19:50,866 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:19:52,012 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:19:56,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:19:58,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:20:01,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:05,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:20:05,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:20:07,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:12,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:13,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:20:17,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:20,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:20:20,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:20:22,012 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:20:23,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:28,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:28,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:20:32,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:20:33,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:35,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:20:35,866 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:20:39,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:43,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:20:44,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:20:44,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:49,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:51,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:20:51,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:20:52,094 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:20:54,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:20:54,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:20:58,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:21:00,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:05,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:06,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:21:06,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:21:06,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:21:11,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:14,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:21:17,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:18,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:21:21,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:21:21,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:21:22,146 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:21:22,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:27,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:28,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:21:32,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:37,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:21:37,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:21:37,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:40,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:21:43,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:48,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:52,015 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:21:52,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:21:52,475 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:21:52,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:21:53,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:21:58,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:02,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:22:04,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:08,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:22:08,359 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:22:10,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:14,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:22:15,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:20,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:22,017 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:22:23,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:22:23,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:22:25,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:26,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:22:31,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:37,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:22:37,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:38,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:22:38,350 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:22:42,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:48,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:22:48,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:52,018 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:22:53,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:22:53,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:22:53,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:22:59,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:00,367 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:04,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:08,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:23:08,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:23:10,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:10,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:15,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:20,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:21,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:22,019 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:23:23,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:23:23,352 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:23:26,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:30,594 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:32,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:37,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:38,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:23:38,495 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:23:40,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:42,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:48,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:50,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:23:52,019 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:23:53,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:23:53,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:23:53,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:23:58,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:00,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:03,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:08,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:24:08,654 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:24:08,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:09,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:24:09,996 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:24:09,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:24:10,000 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:24:10,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:24:10,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:11,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:14,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:19,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:22,020 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:24:22,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:23,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:24:23,653 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:24:24,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:29,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:24:30,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:33,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:36,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:39,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:24:39,835 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:24:41,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:43,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:47,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:52,020 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:24:53,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:24:53,239 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:24:54,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:24:54,834 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:24:58,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:01,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:25:03,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:04,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:08,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:09,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:25:09,836 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:25:13,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:14,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:16,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:25:19,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:22,022 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:25:24,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:24,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:24,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:25:24,835 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:25:30,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:32,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:25:34,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:35,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:39,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:25:39,834 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:25:41,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:44,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:46,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:47,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:25:51,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:25:52,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:25:54,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:25:54,835 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:25:56,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:25:57,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:02,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:02,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:26:06,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:26:07,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:09,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:26:09,835 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:26:13,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:17,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:26:18,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:19,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:26:22,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:26:24,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:25,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:26:25,404 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:26:29,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:26:29,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:35,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:39,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:26:40,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:26:40,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:26:40,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:45,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:47,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:26:51,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:26:51,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:26:52,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:26:55,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:26:55,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:26:56,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:00,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:02,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:07,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:11,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:27:11,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:27:12,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:14,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:18,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:22,025 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:27:24,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:26,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:27:26,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:27:26,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:29,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:35,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:36,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:41,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:41,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:27:41,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:27:46,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:47,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:27:47,172 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:27:47,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:27:47,179 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:27:47,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:27:47,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:27:47,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:48,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:27:52,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:27:52,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:56,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:27:56,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:27:57,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:27:58,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:02,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:28:02,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:07,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:09,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:11,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:28:11,840 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:28:13,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:17,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:28:18,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:21,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:22,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:28:24,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:28,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:28:28,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:28:29,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:31,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:35,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:40,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:43,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:43,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:28:43,423 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:28:45,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:51,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:52,031 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:28:55,445 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:28:56,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:28:58,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:28:58,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:29:01,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:07,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:09,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:29:12,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:13,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:29:13,423 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:29:17,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:18,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:29:22,030 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:29:23,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:28,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:28,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:29:28,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:29:30,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:29:33,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:38,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:40,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:29:43,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:29:43,423 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:29:43,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:49,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:52,030 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:29:52,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:29:55,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:29:58,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:29:58,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:30:00,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:05,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:30:06,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:11,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:13,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:30:13,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:30:16,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:19,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:30:22,033 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:30:22,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:27,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:28,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:30:28,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:30:31,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:30:32,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:37,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:42,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:43,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:30:43,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:30:43,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:30:48,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:52,033 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:30:53,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:30:55,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:30:58,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:30:58,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:30:59,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:05,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:07,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:11,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:13,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:31:13,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:31:16,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:19,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:22,034 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:31:22,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:27,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:28,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:31:28,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:31:30,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:32,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:37,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:40,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:31:40,579 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:31:40,579 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:31:40,581 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:31:40,940 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:31:40,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:42,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:42,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:43,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:31:43,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:31:48,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:52,035 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:31:53,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:31:54,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:31:58,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:31:58,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:31:59,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:04,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:05,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:32:09,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:13,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:32:13,428 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:32:15,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:32:15,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:21,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:22,038 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:32:27,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:32:27,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:28,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:32:28,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:32:32,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:37,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:32:38,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:43,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:43,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:32:43,429 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:32:48,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:49,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:32:52,038 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:32:54,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:58,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:32:58,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:32:59,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:32:59,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:04,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:09,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:11,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:13,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:33:13,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:33:15,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:20,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:22,038 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:33:23,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:26,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:28,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:33:28,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:33:31,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:35,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:36,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:41,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:43,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:33:43,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:33:45,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:47,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:52,041 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:33:53,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:55,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:33:58,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:33:58,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:33:58,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:34:03,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:04,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:34:07,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:34:09,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:13,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:34:13,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:34:14,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:19,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:34:20,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:22,041 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:34:26,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:28,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:34:28,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:34:29,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:34:31,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:37,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:41,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:34:42,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:43,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:34:43,428 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:34:48,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:51,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:34:52,041 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:34:54,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:59,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:34:59,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:34:59,850 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:35:03,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:04,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:09,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:13,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:14,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:35:14,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:35:15,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:20,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:22,043 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:35:23,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:35:23,317 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:35:23,317 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:35:23,319 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:35:23,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:35:24,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:25,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:26,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:29,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:35:29,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:35:32,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:36,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:37,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:42,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:44,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:35:44,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:35:47,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:48,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:52,044 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:35:54,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:57,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:35:59,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:35:59,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:35:59,849 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:36:05,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:05,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:36:09,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:36:11,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:14,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:36:14,852 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:36:16,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:19,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:36:21,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:22,044 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:36:27,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:29,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:36:29,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:36:29,853 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:36:32,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:37,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:42,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:44,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:36:44,850 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:36:47,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:49,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:36:52,046 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:36:53,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:58,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:36:59,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:36:59,850 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:37:01,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:03,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:08,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:11,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:13,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:14,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:37:14,850 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:37:19,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:20,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:37:22,046 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:37:23,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:25,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:30,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:30,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:37:30,460 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:37:34,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:35,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:40,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:44,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:45,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:37:45,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:37:45,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:50,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:37:51,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:37:52,048 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:37:56,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:37:56,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:00,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:38:00,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:38:01,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:06,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:38:06,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:38:06,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:11,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:15,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:38:15,461 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:38:17,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:38:17,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:21,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:38:22,048 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:38:23,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:27,471 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:38:28,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:30,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:38:30,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:38:33,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:36,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:38:37,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:38:38,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:43,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:46,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:38:46,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:38:49,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:49,641 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:38:51,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:38:52,048 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:38:55,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:38:59,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:00,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:01,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:39:01,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:39:06,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:07,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:39:08,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:39:08,926 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:39:08,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:39:08,929 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:39:09,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:39:09,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:10,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:11,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:16,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:39:16,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:39:17,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:21,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:22,049 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:39:22,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:39:22,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:27,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:31,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:39:31,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:39:31,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:33,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:37,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:39:38,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:44,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:44,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:46,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:39:46,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:39:50,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:39:52,049 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:39:52,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:39:54,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:39:55,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:00,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:01,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:40:01,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:40:04,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:06,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:07,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:40:11,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:14,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:16,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:40:16,182 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:40:17,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:22,052 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:40:22,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:40:22,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:26,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:27,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:31,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:40:31,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:40:33,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:36,497 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:37,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:40:38,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:43,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:46,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:40:46,183 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:40:47,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:49,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:52,051 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:40:52,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:40:54,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:40:57,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:40:59,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:01,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:41:01,183 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:41:05,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:07,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:41:07,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:41:10,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:15,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:16,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:41:16,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:41:19,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:41:21,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:22,052 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:41:22,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:41:26,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:29,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:41:31,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:41:31,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:41:32,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:37,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:37,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:41:39,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:41:43,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:46,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:41:46,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:41:48,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:52,053 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:41:52,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:41:52,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:41:53,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:41:58,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:01,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:42:01,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:42:02,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:04,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:07,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:42:09,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:14,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:15,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:16,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:42:16,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:42:21,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:22,055 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:42:22,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:42:24,286 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:26,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:31,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:32,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:42:32,431 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:42:36,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:37,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:37,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:42:42,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:44,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:42:44,527 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:42:44,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:42:44,532 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:42:45,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:42:45,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:46,466 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:47,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:42:47,423 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:42:48,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:52,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:42:53,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:42:53,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:42:56,549 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:42:59,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:02,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:43:02,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:43:04,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:08,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:43:08,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:43:09,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:14,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:17,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:43:17,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:43:18,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:43:20,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:22,056 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:43:23,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:43:25,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:29,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:43:30,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:32,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:43:32,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:43:36,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:38,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:43:39,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:43:41,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:46,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:48,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:43:48,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:43:51,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:43:52,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:43:52,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:43:53,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:43:57,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:02,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:03,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:44:03,169 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:44:03,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:08,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:08,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:44:13,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:14,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:18,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:44:18,183 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:44:19,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:22,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:44:23,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:44:24,294 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:24,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:29,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:33,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:44:33,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:44:34,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:35,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:38,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:44:40,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:44,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:46,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:48,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:44:48,184 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:44:51,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:44:52,071 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:44:53,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:44:56,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:44:56,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:01,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:45:03,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:45:06,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:45:07,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:08,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:45:12,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:18,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:45:18,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:45:18,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:18,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:45:22,059 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:45:24,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:25,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:45:28,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:45:29,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:33,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:45:33,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:45:34,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:39,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:40,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:45:40,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:45:45,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:48,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:45:48,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:45:50,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:45:50,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:45:52,060 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:45:55,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:45:56,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:00,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:01,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:03,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:46:03,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:46:07,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:10,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:46:10,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:13,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:18,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:19,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:46:19,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:46:20,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:46:20,792 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:46:20,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:46:20,796 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:46:21,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:46:21,464 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:22,064 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:46:22,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:24,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:25,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:46:29,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:32,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:34,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:46:34,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:46:34,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:39,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:40,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:46:44,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:45,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:49,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:46:49,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:46:50,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:52,063 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:46:54,715 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:46:55,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:46:55,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:47:00,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:04,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:47:04,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:47:06,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:06,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:47:10,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:47:11,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:16,869 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:47:17,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:19,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:47:19,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:47:22,063 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:47:23,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:25,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:47:26,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:47:28,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:33,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:34,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:47:34,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:47:39,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:47:39,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:40,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:47:44,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:49,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:47:49,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:47:49,770 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:47:50,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:52,063 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:47:56,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:47:57,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:48:01,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:01,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:04,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:48:04,770 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:48:06,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:10,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:12,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:48:12,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:17,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:19,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:48:19,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:48:22,064 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:48:22,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:22,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:27,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:48:27,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:32,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:32,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:34,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:48:34,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:48:37,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:42,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:48:43,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:44,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:48,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:49,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:48:49,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:48:52,071 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:48:54,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:48:54,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:48:57,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:49:00,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:04,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:49:04,778 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:49:05,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:06,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:49:10,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:12,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:49:16,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:16,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:49:19,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:49:19,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:49:21,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:22,066 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:49:27,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:49:27,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:28,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:49:32,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:35,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:49:35,703 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:49:38,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:38,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:49:42,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:49:43,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:48,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:49,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:49:50,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:49:50,701 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:49:52,066 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:49:54,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:49:57,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:49:58,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:49:58,615 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:49:58,615 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:49:58,629 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:49:59,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:49:59,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:00,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:01,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:04,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:05,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:50:05,704 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:50:09,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:11,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:15,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:20,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:21,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:21,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:50:21,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:50:22,159 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:50:26,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:31,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:33,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:36,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:36,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:50:36,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:50:42,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:42,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:50:45,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:47,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:51,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:50:51,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:50:52,115 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:50:53,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:50:54,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:50:57,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:50:58,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:03,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:04,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:51:06,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:51:06,933 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:51:09,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:13,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:51:14,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:16,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:51:19,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:21,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:51:21,935 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:51:22,089 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:51:25,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:26,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:51:28,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:51:30,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:35,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:36,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:51:36,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:51:38,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:51:41,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:43,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:51:46,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:48,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:51:51,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:51,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:51:51,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:51:52,132 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:51:57,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:51:58,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:52:01,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:52:03,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:08,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:08,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:52:08,483 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:52:11,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:52:13,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:18,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:21,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:52:22,071 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:52:23,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:52:23,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:52:23,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:29,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:34,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:38,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:52:38,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:52:39,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:45,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:50,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:52,075 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:52:53,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:52:53,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:52:55,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:52:57,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:53:01,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:06,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:09,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:53:09,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:53:12,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:12,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:53:17,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:22,074 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:53:22,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:53:23,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:24,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:53:24,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:53:28,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:34,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:34,840 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:53:39,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:53:39,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:53:39,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:44,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:53:45,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:50,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:52,075 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:53:54,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:53:54,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:53:55,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:53:56,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:01,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:05,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:54:05,316 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:54:05,317 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:54:05,319 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:54:06,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:54:06,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:07,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:08,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:09,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:54:09,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:54:12,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:14,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 19:54:17,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:19,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:22,076 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:54:23,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:25,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:54:25,588 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:54:28,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:29,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:33,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:39,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:39,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:40,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:54:40,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:54:44,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:49,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:54:50,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:52,077 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:54:56,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:54:57,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:54:57,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:55:01,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:01,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:06,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:11,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:12,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:55:12,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:55:12,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:17,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:22,077 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:55:23,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:23,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:27,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:55:27,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:55:28,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:33,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:34,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:39,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:42,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:55:42,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:55:44,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:45,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:50,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:52,079 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:55:54,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:55:55,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:55:57,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:55:57,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:56:01,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:06,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:07,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:56:11,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:12,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:56:12,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:56:16,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:17,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:56:21,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:22,078 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:56:27,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:56:27,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:56:27,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:29,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:56:32,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:38,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:39,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:56:42,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:56:42,113 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:56:43,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:49,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:49,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:56:52,079 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:56:54,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:56:57,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:56:57,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:56:59,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:01,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:04,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:10,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:11,641 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:12,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:57:12,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:57:16,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:21,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:21,725 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:22,079 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:57:27,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:27,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:57:27,838 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:57:32,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:33,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:37,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:42,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:57:42,839 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:57:43,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 19:57:43,106 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 19:57:43,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 19:57:43,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:43,109 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 19:57:44,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 19:57:44,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:45,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:48,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:52,081 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:57:54,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:57:55,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:57:57,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:57:57,838 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:58:00,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:05,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:58:05,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:10,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:12,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:58:12,841 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:58:15,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:17,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:58:21,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:22,081 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:58:26,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:27,442 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:58:27,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:58:27,839 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:58:32,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:37,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:39,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:58:42,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:58:42,839 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:58:43,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:48,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:51,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:58:52,082 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:58:54,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:59,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:58:59,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:58:59,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:59:03,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:04,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:09,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:13,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:14,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:59:14,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:59:15,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:20,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:22,095 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:59:25,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:25,920 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:29,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:59:29,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:59:31,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:35,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:37,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:42,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:44,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:59:44,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 19:59:46,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:47,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:52,085 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 19:59:53,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:58,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 19:59:58,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 19:59:59,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 19:59:59,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:00:03,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:07,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:00:09,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:14,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:00:14,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:00:14,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:19,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:00:19,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:22,085 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:00:25,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:29,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:00:29,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:00:29,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:00:30,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:35,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:41,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:41,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:00:44,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:00:44,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:00:46,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:51,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:00:51,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:00:52,086 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:00:57,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:00,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:01:00,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:01:02,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:03,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:08,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:13,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:13,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:15,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:01:15,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:01:19,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:22,088 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:01:24,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:01:24,136 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:01:24,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:01:24,139 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:01:24,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:01:25,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:25,868 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:26,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:30,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:30,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:01:30,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:01:35,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:35,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:40,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:45,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:01:45,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:01:46,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:48,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:01:52,088 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:01:52,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:57,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:01:58,105 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:02:00,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:02:00,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:02:02,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:08,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:13,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:15,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:02:15,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:02:18,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:20,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:02:22,088 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:02:24,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:29,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:30,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:02:30,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:02:32,345 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:02:34,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:39,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:42,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:02:44,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:45,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:02:45,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:02:50,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:02:52,089 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:02:53,502 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:02:56,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:01,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:03:01,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:03:01,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:03,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:03:06,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:11,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:15,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:03:16,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:03:16,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:03:17,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:22,090 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:03:23,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:25,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:03:29,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:31,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:03:31,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:03:34,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:37,852 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:03:39,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:44,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:46,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:03:46,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:03:47,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:03:50,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:52,091 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:03:56,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:03:59,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:01,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:04:01,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:04:01,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:07,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:10,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:13,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:16,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:04:16,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:04:18,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:22,091 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:04:22,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:24,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:29,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:31,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:04:31,080 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:04:34,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:34,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:40,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:44,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:45,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:46,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:04:46,065 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:04:51,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:04:52,091 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:04:52,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:04:57,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:01,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:05:01,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:05:01,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:02,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:07,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:11,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:05:11,167 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:05:11,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:05:11,172 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:05:11,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:05:11,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:12,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:13,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:16,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:05:16,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:05:18,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:22,092 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:05:24,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:24,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:29,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:31,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:05:31,067 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:05:34,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:35,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:40,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:44,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:46,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:05:46,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:05:46,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:51,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:05:52,093 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:05:52,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:05:54,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:05:57,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:01,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:06:01,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:06:03,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:04,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:06,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:06:09,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:14,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:15,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:16,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:06:16,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:06:20,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:21,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:06:22,104 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:06:23,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:26,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:31,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:31,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:06:31,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:06:31,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:36,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:06:36,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:41,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:41,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:46,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:06:46,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:06:46,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:51,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:06:51,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:06:52,094 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:06:52,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:06:57,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:01,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:07:01,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:07:01,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:02,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:06,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:07:08,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:12,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:14,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:16,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:07:16,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:07:19,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:20,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:21,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:07:22,095 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:07:25,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:30,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:30,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:31,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:07:31,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:07:36,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:07:36,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:40,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:42,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:47,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:47,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:07:47,223 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:07:50,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:07:52,095 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:07:52,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:57,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:07:58,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:02,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:08:02,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:08:02,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:07,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:08,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:12,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:17,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:08:17,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:08:18,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:18,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:22,096 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:08:24,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:26,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:08:26,327 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:08:26,329 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:08:26,331 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:08:26,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:08:28,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:29,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:32,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:08:32,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:08:34,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:36,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:40,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:45,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:46,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:08:47,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:08:47,254 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:08:51,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:08:52,096 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:08:57,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:00,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:02,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:09:02,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:09:02,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:08,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:10,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:14,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:17,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:09:17,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:09:19,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:19,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:22,097 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:09:25,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:30,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:30,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:32,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:09:32,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:09:36,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:38,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:41,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:46,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:47,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:09:47,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:09:48,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:09:52,098 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:09:52,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:58,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:09:58,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:02,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:10:02,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:10:03,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:06,292 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:09,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:14,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:16,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:17,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:10:17,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:10:19,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:22,099 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:10:25,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:26,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:30,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:32,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:10:32,783 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:10:35,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:36,521 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:41,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:44,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:46,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:47,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:10:47,782 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:10:51,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:10:52,100 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:10:54,661 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:10:56,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:01,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:02,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:11:02,782 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:11:04,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:07,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:12,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:12,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:17,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:11:17,783 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:11:17,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:22,101 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:11:22,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:23,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:28,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:32,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:11:32,783 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:11:32,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:33,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:39,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:39,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:11:39,656 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:11:39,657 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:11:39,658 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:11:40,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:11:41,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:44,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:48,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:11:48,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:11:50,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:11:51,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:11:52,101 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:11:56,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:00,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:01,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:03,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:12:03,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:12:06,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:08,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:11,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:16,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:18,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:12:18,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:12:18,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:22,101 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:12:22,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:27,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:28,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:32,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:33,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:12:33,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:12:38,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:38,447 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:43,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:48,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:12:48,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:12:48,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:48,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:52,102 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:12:54,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:12:56,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:12:59,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:03,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:13:03,712 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:13:04,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:06,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:10,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:16,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:16,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:18,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:13:18,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:13:21,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:22,103 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:13:26,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:26,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:31,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:33,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:13:33,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:13:35,051 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:37,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:42,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:45,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:48,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:48,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:13:48,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:13:52,104 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:13:53,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:13:54,330 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:13:58,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:03,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:03,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:14:03,706 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:14:04,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:08,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:13,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:14,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:18,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:14:18,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:14:18,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:22,104 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:14:22,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:24,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:29,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:32,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:33,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:14:33,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:14:34,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:40,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:42,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:46,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:49,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:14:49,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:14:51,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:14:51,148 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:14:51,149 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:14:51,152 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:14:51,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:14:51,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:14:52,105 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:14:52,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:53,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:14:57,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:02,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:02,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:15:04,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:15:04,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:15:07,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:12,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:17,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:19,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:15:19,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:15:22,107 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:15:23,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:26,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:15:28,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:33,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:34,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:15:34,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:15:37,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:15:39,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:45,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:15:45,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:49,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:15:49,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:15:50,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:52,108 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:15:56,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:15:57,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:01,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:04,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:16:04,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:16:06,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:07,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:12,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:15,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:17,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:19,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:16:19,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:16:22,110 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:16:23,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:24,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:28,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:33,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:34,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:34,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:16:34,768 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:16:39,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:44,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:46,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:16:49,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:16:49,768 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:16:50,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:52,111 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:16:55,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:16:56,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:00,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:04,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:17:04,769 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:17:05,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:06,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:11,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:16,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:17,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:19,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:17:19,768 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:17:22,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:22,111 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:17:25,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:17:27,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:29,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:32,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:36,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:17:36,275 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:17:37,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:39,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:43,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:49,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:49,385 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:17:51,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:17:51,272 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:17:52,112 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:17:54,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:57,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:17:59,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:17:59,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:04,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:07,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:18:07,265 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:18:09,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:11,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:15,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:20,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:21,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:22,112 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:18:22,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:18:22,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:18:26,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:31,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:18:31,080 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:18:31,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:18:31,084 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:18:31,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:18:32,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:32,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:33,733 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:37,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:37,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:18:37,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:18:42,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:43,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:48,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:52,115 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:18:52,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:18:52,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:18:52,905 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:18:53,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:18:59,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:02,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:04,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:07,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:19:07,289 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:19:09,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:14,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:15,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:19,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:22,115 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:19:22,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:19:22,474 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:19:25,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:27,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:31,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:36,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:37,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:37,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:19:37,470 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:19:42,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:47,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:47,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:52,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:19:52,116 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:19:52,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:19:52,470 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:19:57,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:19:58,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:03,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:07,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:20:07,470 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:20:08,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:09,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:20:14,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:19,698 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:20:19,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:22,116 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:20:22,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:20:22,470 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:20:25,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:31,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:31,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:20:36,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:37,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:20:37,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:20:40,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:20:42,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:47,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:50,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:20:52,117 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:20:52,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:20:52,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:20:52,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:57,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:20:59,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:21:01,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:03,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:07,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:21:07,532 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:21:08,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:13,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:14,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:21:14,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:19,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:21,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:22,117 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:21:22,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:21:22,530 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:21:24,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:29,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:21:30,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:31,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:35,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:37,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:21:37,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:21:41,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:41,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:47,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:51,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:21:52,118 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:21:52,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:52,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:21:52,783 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:21:57,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:21:57,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:21:57,678 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:21:57,678 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:21:57,680 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:21:58,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:21:59,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:03,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:07,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:22:07,783 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:22:08,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:09,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:13,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:14,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:22:19,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:19,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:22,118 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:22:22,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:22:22,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:22:24,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:29,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:30,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:35,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:37,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:22:37,784 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:22:39,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:40,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:44,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:22:46,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:47,793 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:51,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:52,120 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:22:53,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:22:53,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:22:57,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:22:57,854 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:22:59,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:23:02,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:05,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:08,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:08,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:23:08,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:23:14,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:14,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:23:15,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:19,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:22,122 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:23:23,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:23:23,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:23:24,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:26,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:29,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:23:30,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:33,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:36,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:38,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:23:38,775 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:23:41,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:43,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:44,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:23:46,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:52,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:52,122 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:23:53,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:23:53,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:23:53,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:23:57,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:23:59,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:24:03,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:07,337 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:08,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:24:08,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:24:08,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:13,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:14,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:24:15,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:19,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:22,123 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:24:23,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:24:23,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:24:24,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:25,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:29,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:24:30,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:35,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:36,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:38,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:24:38,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:24:42,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:44,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:24:45,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:47,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:52,124 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:24:53,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:24:53,775 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:24:53,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:24:55,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:24:58,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:00,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:25:03,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:04,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:08,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:25:08,776 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:25:10,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:11,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:25:11,893 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:25:11,894 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:25:11,897 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:25:12,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:25:13,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:15,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:15,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:25:20,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:22,124 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:25:23,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:25:23,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:25:23,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:26,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:30,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:25:31,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:33,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:36,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:39,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:25:39,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:25:41,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:42,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:45,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:25:46,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:52,125 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:25:52,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:25:52,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:25:54,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:25:54,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:25:57,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:00,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:26:02,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:02,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:07,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:09,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:26:09,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:26:12,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:13,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:17,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:26:19,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:19,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:22,125 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:26:24,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:26:24,681 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:26:24,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:29,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:30,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:32,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:26:36,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:39,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:39,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:26:39,680 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:26:41,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:46,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:47,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:26:49,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:26:52,136 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:26:52,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:55,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:26:55,374 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:26:57,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:26:57,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:02,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:27:03,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:07,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:08,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:10,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:27:10,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:27:13,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:15,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:17,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:27:19,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:22,128 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:27:24,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:25,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:27:25,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:27:25,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:29,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:33,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:27:35,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:36,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:40,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:27:40,372 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:27:40,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:45,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:46,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:48,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:27:51,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:27:52,130 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:27:54,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:27:55,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:27:55,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:27:56,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:01,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:03,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:28:04,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:07,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:11,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:28:11,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:28:12,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:14,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:17,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:18,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:28:22,130 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:28:22,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:28:22,216 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:28:22,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:28:22,218 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:28:22,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:28:23,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:24,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:26,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:28:26,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:28:28,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:33,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:28:34,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:34,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:39,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:41,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:28:41,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:28:41,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:44,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:48,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:28:50,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:51,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:28:52,130 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:28:56,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:28:56,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:28:56,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:29:01,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:01,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:03,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:29:06,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:09,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:11,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:29:11,225 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:29:11,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:16,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:18,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:29:19,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:22,131 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:29:22,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:26,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:29:26,617 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:29:27,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:29,841 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:32,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:33,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:29:38,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:39,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:41,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:29:41,615 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:29:43,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:47,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:29:48,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:29:49,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:52,131 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:29:55,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:29:56,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:29:56,615 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:29:58,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:00,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:04,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:30:06,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:08,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:11,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:11,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:30:11,616 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:30:17,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:18,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:19,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:30:22,133 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:30:23,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:26,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:26,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:30:26,616 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:30:28,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:34,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:30:34,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:36,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:39,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:41,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:30:41,844 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:30:44,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:45,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:49,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:30:50,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:52,133 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:30:54,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:30:55,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:30:56,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:30:56,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:31:01,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:04,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:31:04,539 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:06,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:11,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:11,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:31:11,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:31:14,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:17,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:20,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:31:22,133 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:31:22,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:23,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:26,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:31:26,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:31:27,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:31,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:31:31,723 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:31:31,724 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:31:31,726 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:31:31,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:31:32,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:33,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:33,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:35,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:31:39,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:41,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:31:41,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:31:43,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:45,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:50,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:50,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:31:51,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:31:52,134 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:31:55,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:31:56,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:31:56,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:32:00,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:02,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:05,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:32:06,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:11,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:11,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:32:11,843 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:32:12,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:17,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:20,149 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:20,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:32:22,137 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:32:22,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:27,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:27,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:32:27,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:32:30,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:32,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:35,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:32:37,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:40,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:42,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:32:42,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:32:43,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:48,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:50,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:50,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:32:52,135 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:32:54,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:32:57,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:32:57,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:32:58,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:32:59,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:04,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:05,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:33:08,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:09,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:12,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:33:12,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:33:15,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:18,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:20,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:33:20,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:22,136 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:33:26,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:26,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:27,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:33:27,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:33:32,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:35,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:33:36,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:37,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:42,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:43,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:33:43,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:33:46,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:47,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:50,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:33:52,137 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:33:53,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:56,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:33:58,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:33:58,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:33:58,804 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:34:03,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:03,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:05,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:34:08,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:13,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:34:13,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:34:14,007 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:14,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:19,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:21,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:34:22,138 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:34:24,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:24,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:28,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:34:28,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:34:29,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:34,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:35,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:41,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:41,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:34:41,957 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:34:41,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:34:41,960 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:34:42,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:34:42,189 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:43,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:34:43,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:34:44,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:47,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:52,138 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:34:52,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:52,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:34:57,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:34:59,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:34:59,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:35:02,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:02,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:07,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:12,405 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:12,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:14,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:35:14,049 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:35:18,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:22,139 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:35:22,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:24,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:29,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:30,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:35:30,478 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:35:34,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:34,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:40,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:44,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:45,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:35:45,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:35:45,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:50,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:35:51,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:35:52,144 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:35:54,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:35:56,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:01,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:02,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:36:02,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:36:04,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:06,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:11,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:14,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:16,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:17,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:36:17,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:36:22,141 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:36:22,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:25,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:27,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:32,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:36:32,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:36:33,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:35,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:38,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:43,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:45,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:47,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:36:47,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:36:49,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:36:52,142 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:36:54,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:36:55,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:00,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:02,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:37:02,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:37:04,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:05,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:10,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:12,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:15,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:17,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:37:17,038 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:37:21,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:22,142 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:37:24,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:27,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:32,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:37:32,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:37:32,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:34,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:37,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:43,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:44,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:47,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:37:47,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:37:49,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:52,143 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:37:54,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:37:54,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:37:59,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:02,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:38:02,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:38:05,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:07,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:38:07,249 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:38:07,251 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:38:07,256 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:38:07,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:38:08,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:10,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:15,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:17,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:38:17,040 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:38:18,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:20,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:22,143 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:38:26,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:28,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:31,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:32,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:38:32,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:38:36,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:39,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:41,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:47,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:38:47,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:38:47,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:49,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:52,144 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:38:53,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:38:57,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:38:59,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:02,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:39:02,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:39:04,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:09,239 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:39:09,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:14,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:17,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:39:17,040 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:39:19,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:39:20,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:22,144 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:39:26,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:30,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:39:31,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:32,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:39:32,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:39:36,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:37,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:39:40,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:39:41,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:46,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:47,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:39:47,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:39:50,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:39:51,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:39:52,147 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:39:57,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:00,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:02,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:40:02,432 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:40:02,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:12,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:13,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:17,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:40:17,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:40:18,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:22,176 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:40:22,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:24,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:29,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:32,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:40:32,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:40:32,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:34,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:40,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:42,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:46,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:47,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:40:47,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:40:51,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:40:52,146 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:40:55,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:40:57,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:02,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:02,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:41:02,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:41:05,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:08,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:14,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:15,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:17,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:41:17,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:41:19,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:22,147 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:41:23,330 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:25,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:30,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:32,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:41:32,426 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:41:32,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:41:32,429 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:41:32,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:41:32,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:41:33,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:41:33,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:35,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:35,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:41,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:45,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:47,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:47,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:41:47,433 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:41:52,148 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:41:52,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:41:54,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:41:57,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:02,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:42:02,539 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:42:02,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:04,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:42:08,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:13,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:14,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:42:17,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:42:17,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:42:18,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:22,150 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:42:23,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:24,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:42:28,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:33,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:42:33,111 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:42:34,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:36,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:42:39,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:44,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:47,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:42:48,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:42:48,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:42:50,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:52,150 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:42:55,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:42:57,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:00,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:03,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:43:03,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:43:06,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:09,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:11,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:16,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:18,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:43:18,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:43:19,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:22,151 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:43:22,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:27,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:29,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:32,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:33,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:43:33,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:43:38,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:39,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:43,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:48,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:43:48,109 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:43:49,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:43:49,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:43:52,152 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:43:54,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:00,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:00,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:03,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:44:03,109 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:44:05,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:08,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:44:10,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:10,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:15,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:19,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:44:19,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:44:20,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:22,152 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:44:23,001 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:26,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:31,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:34,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:44:34,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:44:35,103 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:36,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:42,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:45,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:47,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:49,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:44:49,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:44:52,155 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:44:53,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:44:55,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:44:58,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:03,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:04,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:45:04,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:45:05,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:45:05,262 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:45:05,263 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:45:05,267 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:45:05,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:45:07,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:45:09,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:14,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:17,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:45:19,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:19,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:45:19,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:45:22,155 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:45:25,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:29,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:45:30,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:34,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:45:34,621 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:45:35,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:39,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:45:41,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:46,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:49,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:45:49,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:45:51,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:45:51,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:45:52,155 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:45:57,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:01,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:46:02,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:04,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:46:04,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:46:07,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:10,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:46:13,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:18,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:19,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:46:19,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:46:22,156 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:46:23,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:46:24,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:29,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:33,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:46:34,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:34,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:46:34,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:46:40,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:45,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:46:45,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:49,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:46:49,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:46:51,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:52,158 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:46:57,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:46:57,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:02,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:04,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:47:04,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:47:07,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:07,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:13,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:17,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:19,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:19,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:47:19,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:47:22,169 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:47:25,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:25,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:47:29,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:31,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:34,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:47:34,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:47:37,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:39,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:42,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:47,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:49,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:47:49,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:47:49,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:47:52,158 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:47:53,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:47:55,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:47:58,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:01,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:03,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:04,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:48:04,847 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:48:09,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:12,082 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:14,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:19,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:20,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:48:20,643 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:48:22,159 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:48:23,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:25,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:25,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:48:30,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:35,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:35,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:48:35,641 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:48:35,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:40,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:42,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:48:43,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:48:43,978 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:48:43,979 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:48:44,000 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:48:44,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:48:45,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:46,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:50,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:48:50,641 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:48:51,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:52,163 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:48:55,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:48:57,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:48:57,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:49:02,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:05,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:49:05,642 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:49:07,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:49:08,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:12,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:49:13,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:19,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:19,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:49:20,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:49:20,641 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:49:22,161 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:49:25,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:27,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:49:29,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:49:30,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:35,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:35,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:49:35,644 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:49:40,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:49:41,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:42,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:49:46,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:50,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:49:50,642 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:49:51,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:52,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:49:52,162 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:49:57,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:49:57,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:50:02,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:02,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:05,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:50:05,642 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:50:07,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:12,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:50:13,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:13,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:18,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:21,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:50:21,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:50:22,164 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:50:23,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:25,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:27,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:50:29,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:34,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:35,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:36,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:50:36,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:50:39,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:43,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:50:45,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:45,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:50,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:51,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:50:51,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:50:52,165 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:50:56,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:50:57,793 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:50:58,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:51:02,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:06,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:51:06,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:51:07,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:09,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:51:13,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:51:13,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:18,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:21,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:51:21,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:51:22,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:51:22,165 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:51:23,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:28,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:28,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:51:32,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:51:33,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:36,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:51:36,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:51:38,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:43,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:51:44,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:51:44,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:49,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:51,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:51:51,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:51:52,166 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:51:54,345 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:51:55,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:51:58,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:52:00,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:04,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:05,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:06,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:52:06,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:52:11,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:13,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:52:15,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:16,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:21,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:22,167 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:52:22,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:52:22,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:52:23,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:52:23,825 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:52:23,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:52:23,828 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:52:24,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:52:25,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:27,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:28,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:52:32,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:35,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:37,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:52:37,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:52:38,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:43,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:43,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:52:45,793 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:49,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:52,169 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:52:52,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:52:52,820 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:52:55,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:52:55,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:52:58,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:53:00,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:05,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:53:06,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:07,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:53:07,819 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:53:11,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:13,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:53:16,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:53:17,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:22,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:53:22,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:53:22,910 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:53:23,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:26,089 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:53:28,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:53:28,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:33,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:36,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:53:37,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:53:37,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:53:39,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:43,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:53:44,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:50,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:53:50,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:52,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:53:52,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:53:52,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:53:56,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:53:58,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:54:01,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:02,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:06,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:08,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:54:08,817 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:54:12,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:12,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:17,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:22,182 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:54:22,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:23,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:23,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:54:23,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:54:29,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:29,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:54:32,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:34,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:39,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:40,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:54:40,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:54:41,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:45,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:50,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:54:52,172 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:54:53,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:54:55,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:54:55,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:54:56,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:01,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:03,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:55:06,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:10,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:55:10,092 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:55:12,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:14,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:55:17,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:22,173 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:55:23,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:25,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:55:25,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:55:26,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:55:28,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:33,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:36,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:55:39,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:40,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:55:40,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:55:44,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:46,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:55:49,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:52,175 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:55:53,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:55:53,876 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:55:53,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:55:53,879 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:55:54,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:55:54,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:55:55,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:55:55,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:55:56,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:00,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:05,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:06,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:10,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:56:10,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:56:11,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:14,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:16,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:21,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:22,175 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:56:24,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:25,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:56:25,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:56:26,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:31,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:56:32,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:34,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:37,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:40,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:56:40,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:56:42,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:44,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:46,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:56:48,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:52,176 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:56:53,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:56:53,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:56:55,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:56:55,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:56:58,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:01,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:57:03,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:04,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:09,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:10,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:57:10,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:57:12,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:15,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:16,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:57:20,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:22,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:22,176 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:57:25,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:57:25,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:57:26,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:31,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:57:32,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:32,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:37,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:40,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:57:40,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:57:42,286 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:42,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:46,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:57:48,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:52,177 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:57:52,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:57:53,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:57:55,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:57:55,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:57:58,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:01,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:58:02,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:04,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:09,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:10,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:58:10,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:58:12,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:14,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:16,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:58:20,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:22,177 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:58:22,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:25,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:25,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:58:25,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:58:30,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:31,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:58:32,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:36,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:40,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:58:40,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:58:40,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:41,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:46,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:47,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:58:50,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:58:52,178 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:58:52,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:56,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:58:56,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:58:57,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:58:58,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:02,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 20:59:03,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 20:59:06,916 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 20:59:06,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 20:59:06,920 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 20:59:06,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 20:59:07,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:08,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:08,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:11,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:59:11,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:59:14,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:19,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:20,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:22,179 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:59:25,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:26,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:59:26,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:59:28,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:30,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:35,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:38,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:40,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:41,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:59:41,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:59:46,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 20:59:46,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:52,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:52,180 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 20:59:56,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 20:59:56,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 20:59:57,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 20:59:58,352 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:03,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:08,438 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:08,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:11,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:00:11,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:00:14,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:16,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:19,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:20,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:22,180 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:00:25,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:26,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:00:26,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:00:28,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:30,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:36,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:38,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:41,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:00:41,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:00:41,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:46,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:48,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:00:52,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:52,181 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:00:56,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:00:56,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:00:57,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:00:58,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:02,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:01:03,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:06,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:08,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:11,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:01:11,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:01:14,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:16,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:20,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:22,182 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:01:26,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:01:26,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:01:26,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:27,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:31,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:33,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:01:36,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:37,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:41,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:01:41,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:01:42,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:47,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:47,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:48,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:01:52,183 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:01:53,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:01:56,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:01:56,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:01:57,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:01:58,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:03,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:03,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:02:06,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:08,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:11,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:02:11,140 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:02:14,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:16,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:18,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:02:19,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:22,185 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:02:24,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:02:24,086 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:02:24,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:02:24,089 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:02:24,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:02:25,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:26,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:02:26,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:02:26,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:30,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:33,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:02:35,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:36,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:40,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:41,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:02:41,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:02:46,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:46,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:48,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:02:51,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:02:52,186 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:02:56,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:02:56,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:02:56,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:02:57,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:02,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:03,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:03:06,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:07,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:11,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:03:11,139 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:03:13,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:14,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:18,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:03:18,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:22,188 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:03:23,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:25,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:26,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:03:26,138 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:03:29,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:33,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:03:34,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:35,103 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:39,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:41,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:41,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:03:41,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:03:45,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:46,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:47,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:03:48,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:03:50,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:52,188 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:03:56,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:03:56,138 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:03:56,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:03:57,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:01,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:03,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:04:06,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:07,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:11,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:04:11,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:04:12,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:15,405 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:18,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:18,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:04:22,190 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:04:23,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:25,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:26,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:04:26,138 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:04:28,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:33,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:04:33,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:34,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:38,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:41,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:04:41,655 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:04:43,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:44,636 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:48,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:04:49,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:52,192 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:04:54,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:04:55,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:04:56,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:04:56,654 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:05:00,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:03,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:05:04,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:06,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:11,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:12,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:05:12,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:05:14,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:16,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:18,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:05:21,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:22,193 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:05:27,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:27,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:27,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:05:27,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:05:33,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:33,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:05:34,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:05:34,956 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:05:34,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:05:34,970 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:05:35,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:05:37,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:38,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:43,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:05:43,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:05:43,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:47,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:49,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:52,194 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:05:55,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:05:57,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:05:58,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:05:58,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:06:00,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:04,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:06:05,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:07,371 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:10,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:14,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:06:14,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:06:16,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:17,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:19,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:06:21,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:22,195 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:06:27,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:27,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:29,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:06:29,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:06:33,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:36,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:38,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:43,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:44,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:06:44,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:06:46,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:49,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:52,196 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:06:54,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:06:55,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:06:59,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:06:59,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:07:00,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:04,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:05,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:10,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:14,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:07:14,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:07:14,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:16,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:21,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:21,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:07:22,197 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:07:25,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:26,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:29,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:07:29,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:07:32,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:33,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:36,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:07:37,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:43,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:43,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:44,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:07:44,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:07:49,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:51,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:07:52,197 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:07:53,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:07:55,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:07:59,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:07:59,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:08:00,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:03,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:05,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:06,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:08:11,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:13,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:14,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:08:14,701 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:08:16,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:21,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:08:22,199 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:08:22,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:27,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:27,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:29,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:08:29,701 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:08:32,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:36,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:08:37,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:38,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:43,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:44,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:08:44,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:08:47,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:48,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:51,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:08:52,199 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:08:54,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:08:55,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:08:55,796 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:08:55,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:08:55,800 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:08:56,787 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:08:56,788 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:57,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:08:59,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:08:59,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:08:59,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:04,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:06,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:09:06,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:10,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:14,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:09:14,703 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:09:15,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:17,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:20,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:21,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:09:22,202 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:09:26,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:27,200 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:29,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:09:29,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:09:31,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:36,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:09:37,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:37,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:42,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:44,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:09:44,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:09:45,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:47,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:52,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:09:52,203 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:09:53,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:55,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:09:58,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:09:59,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:09:59,704 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:10:03,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:05,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:07,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:10:09,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:13,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:14,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:14,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:10:14,702 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:10:19,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:22,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:10:22,203 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:10:23,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:25,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:30,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:10:30,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:10:30,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:33,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:36,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:42,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:43,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:45,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:10:45,109 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:10:47,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:52,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:10:52,205 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:10:53,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:10:53,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:10:58,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:00,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:11:00,111 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:11:03,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:03,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:11:07,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:11:09,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:13,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:11:14,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:15,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:11:15,109 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:11:19,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:22,205 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:11:25,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:30,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:11:30,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:11:30,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:33,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:11:36,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:37,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:11:42,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:43,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:11:45,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:11:45,112 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:11:47,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:52,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:11:52,206 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:11:53,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:11:53,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:11:58,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:00,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:12:00,455 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:12:03,371 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:03,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:09,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:11,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:14,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:15,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:12:15,454 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:12:20,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:20,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:12:20,087 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:12:20,088 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:12:20,091 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:12:20,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:12:21,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:22,207 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:12:25,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:30,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:12:30,453 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:12:30,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:31,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:35,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:38,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:12:41,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:41,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:45,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:12:45,453 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:12:46,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:51,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:12:52,208 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:12:52,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:12:53,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:12:58,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:00,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:13:00,454 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:13:01,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:03,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:08,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:13:09,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:11,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:14,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:15,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:13:15,454 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:13:20,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:21,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:22,209 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:13:23,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:13:25,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:30,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:30,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:13:30,454 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:13:30,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:35,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:40,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:41,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:46,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:46,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:13:46,606 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:13:50,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:13:51,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:13:52,209 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:13:57,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:00,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:01,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:14:01,604 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:14:02,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:08,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:09,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:13,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:16,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:14:16,605 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:14:19,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:19,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:22,210 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:14:24,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:27,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:29,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:31,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:14:31,604 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:14:35,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:37,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:41,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:46,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:14:46,605 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:14:46,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:47,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:51,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:14:52,212 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:14:53,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:14:55,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:14:57,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:02,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:15:02,740 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:15:02,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:05,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:08,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:13,731 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:13,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:17,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:15:17,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:15:19,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:22,212 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:15:23,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:24,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:29,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:32,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:15:32,597 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:15:32,597 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:15:32,599 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:15:32,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:15:32,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:15:32,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:15:33,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:35,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:40,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:43,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:45,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:47,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:15:47,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:15:51,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:15:52,213 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:15:54,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:15:56,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:01,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:02,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:16:02,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:16:04,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:07,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:13,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:17,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:17,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:16:17,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:16:22,213 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:16:23,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:23,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:28,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:32,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:16:32,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:16:33,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:34,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:39,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:43,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:45,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:47,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:16:47,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:16:50,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:16:52,213 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:16:53,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:16:56,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:01,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:02,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:02,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:17:02,740 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:17:07,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:09,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:17:11,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:13,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:18,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:17:18,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:17:18,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:20,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:22,214 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:17:24,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:24,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:17:29,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:30,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:33,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:17:33,254 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:17:35,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:40,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:41,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:46,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:48,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:17:48,254 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:17:50,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:17:51,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:52,215 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:17:55,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:17:56,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:17:58,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:01,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:03,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:18:03,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:18:07,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:08,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:10,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:18:12,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:18,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:18:18,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:18:18,445 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:18,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:22,216 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:18:24,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:25,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:18:28,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:29,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:33,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:18:33,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:18:34,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:38,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:39,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:44,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:46,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:18:46,746 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:18:46,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:18:46,748 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:18:47,679 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:18:47,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:18:48,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:18:48,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:18:50,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:52,216 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:18:55,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:18:57,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:01,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:03,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:19:03,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:19:07,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:07,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:17,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:17,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:18,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:19:18,265 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:19:22,217 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:19:23,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:27,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:28,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:33,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:34,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:19:34,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:19:38,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:38,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:44,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:48,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:19:49,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:19:49,448 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:19:50,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:52,217 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:19:55,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:19:58,189 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:20:01,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:04,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:20:04,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:20:06,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:08,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:20:11,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:17,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:18,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:20:19,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:20:19,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:20:22,218 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:20:23,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:25,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:20:28,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:20:28,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:33,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:34,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:20:34,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:20:36,470 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:20:39,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:44,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:49,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:20:49,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:20:49,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:20:52,227 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:20:55,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:00,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:04,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:21:04,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:21:06,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:11,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:12,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:21:14,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:21:16,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:19,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:21:19,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:21:22,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:22,241 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:21:24,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:21:27,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:21:28,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:33,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:33,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:21:34,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:21:34,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:21:38,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:42,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:21:43,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:44,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:21:48,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:49,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:21:49,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:21:52,230 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:21:54,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:21:54,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:21:57,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:22:00,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:04,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:04,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:22:04,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:22:06,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:11,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:12,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:22:14,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:16,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:21,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:22:21,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:22:21,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:22,231 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:22:24,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:26,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:27,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:22:32,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:32,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:22:32,516 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:22:32,517 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:22:32,519 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:22:33,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:22:34,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:36,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:22:36,393 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:22:37,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:42,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:22:43,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:44,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:48,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:51,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:22:51,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:22:52,231 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:22:54,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:22:54,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:22:57,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:23:00,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:02,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:05,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:06,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:23:06,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:23:10,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:12,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:23:12,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:16,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:21,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:21,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:23:21,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:23:22,234 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:23:26,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:26,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:27,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:23:32,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:36,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:23:36,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:23:36,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:37,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:42,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:43,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:23:46,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:48,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:51,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:23:51,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:23:52,239 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:23:53,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:56,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:23:58,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:23:58,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:24:03,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:06,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:24:06,394 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:24:07,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:09,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:13,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:24:15,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:16,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:20,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:22,237 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:24:22,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:24:22,953 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:24:25,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:26,311 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:31,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:36,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:37,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:37,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:24:37,951 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:24:43,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:44,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:24:46,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:48,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:52,239 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:24:53,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:24:53,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:24:54,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:24:56,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:24:59,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:24:59,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:04,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:06,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:08,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:25:08,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:25:10,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:14,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:25:15,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:16,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:20,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:22,241 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:25:24,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:25:24,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:25:26,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:27,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:29,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:25:32,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:37,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:38,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:39,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:25:39,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:25:43,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:44,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:25:47,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:48,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:52,242 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:25:54,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:25:54,145 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:25:54,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:25:54,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:25:54,836 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:25:54,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:25:54,839 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:25:55,220 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:25:56,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:57,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:25:59,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:26:00,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:05,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:06,294 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:26:09,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:26:09,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:26:11,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:14,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:26:14,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:26:17,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:22,243 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:26:23,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:24,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:26:24,145 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:26:26,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:26:28,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:33,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:34,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:26:38,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:39,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:26:39,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:26:44,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:44,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:26:49,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:52,244 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:26:54,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:26:54,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:26:54,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:26:54,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:00,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:04,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:05,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:09,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:27:09,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:27:11,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:15,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:17,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:22,244 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:27:22,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:24,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:27:24,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:27:25,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:28,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:31,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:27:33,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:34,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:39,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:40,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:27:40,526 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:27:43,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:44,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:50,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:52,245 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:27:53,404 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:27:55,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:27:55,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:27:55,525 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:28:00,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:01,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:28:02,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:06,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:10,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:28:10,525 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:28:12,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:12,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:18,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:22,246 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:28:22,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:23,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:25,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:28:25,525 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:28:28,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:32,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:34,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:39,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:40,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:28:40,526 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:28:42,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:44,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:50,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:52,246 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:28:52,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:28:55,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:28:55,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:28:55,525 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:29:00,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:01,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:29:03,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:05,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:10,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:29:10,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:29:11,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:11,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:29:11,658 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:29:11,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:29:11,662 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:29:12,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:29:13,069 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:14,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:17,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:22,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:22,247 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:29:25,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:25,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:29:25,526 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:29:27,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:31,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:29:33,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:33,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:38,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:40,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:29:40,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:29:43,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:43,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:49,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:52,256 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:29:53,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:29:54,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:29:55,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:29:55,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:29:59,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:01,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:30:03,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:05,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:10,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:30:10,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:30:10,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:13,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:16,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:21,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:22,256 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:30:23,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:25,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:30:25,771 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:30:26,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:31,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:30:32,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:32,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:37,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:40,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:30:40,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:30:42,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:42,836 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:48,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:52,257 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:30:52,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:30:53,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:30:55,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:30:55,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:30:59,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:01,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:31:04,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:05,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:09,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:11,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:31:11,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:31:15,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:15,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:21,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:22,259 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:31:26,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:26,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:31:26,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:31:27,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:32,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:37,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:38,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:41,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:31:41,894 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:31:43,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:47,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:48,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:52,259 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:31:54,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:31:56,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:31:56,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:31:59,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:31:59,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:05,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:09,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:32:10,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:11,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:32:11,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:32:16,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:17,666 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:32:21,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:22,260 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:32:26,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:32:26,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:32:27,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:27,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:32:32,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:37,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:37,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:32:37,266 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:32:37,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:32:37,269 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:32:37,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:32:38,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:32:41,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:32:41,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:32:43,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:46,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:32:48,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:52,260 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:32:54,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:32:56,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:32:56,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:32:57,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:00,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:05,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:09,083 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:10,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:11,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:33:11,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:33:16,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:19,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:21,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:22,261 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:33:26,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:33:26,898 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:33:27,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:29,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:32,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:37,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:39,371 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:41,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:33:41,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:33:43,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:49,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:49,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:33:52,262 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:33:54,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:56,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:33:56,897 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:33:59,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:33:59,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:04,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:07,555 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:09,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:11,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:34:11,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:34:15,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:17,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:20,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:22,263 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:34:26,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:26,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:34:26,896 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:34:27,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:31,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:33,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:34:35,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:36,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:41,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:42,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:34:42,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:34:45,817 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:46,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:48,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:34:52,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:34:52,264 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:34:55,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:34:57,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:34:57,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:34:57,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:02,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:03,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:35:05,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:08,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:12,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:35:12,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:35:13,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:14,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:18,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:35:19,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:22,264 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:35:24,069 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:24,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:27,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:35:27,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:35:29,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:31,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:33,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:35:35,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:40,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:42,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:35:42,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:35:43,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:45,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:49,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:35:50,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:50,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:35:50,436 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:35:50,437 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:35:50,438 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:35:50,594 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:35:51,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:52,264 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:35:52,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:35:56,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:35:57,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:35:57,189 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:36:01,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:01,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:04,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:36:07,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:11,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:12,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:36:12,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:36:12,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:17,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:19,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:36:21,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:22,265 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:36:23,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:27,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:36:27,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:36:28,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:29,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:33,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:34,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:36:38,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:39,947 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:42,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:36:42,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:36:44,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:50,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:36:50,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:52,266 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:36:56,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:36:57,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:36:57,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:37:00,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:01,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:05,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:37:06,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:10,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:12,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:12,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:37:12,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:37:18,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:20,239 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:22,266 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:37:23,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:27,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:37:27,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:37:28,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:29,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:33,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:35,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:37:39,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:39,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:42,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:37:42,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:37:44,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:49,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:37:50,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:52,266 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:37:55,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:37:57,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:37:57,528 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:38:00,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:03,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:05,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:10,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:12,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:38:12,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:38:13,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:16,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:21,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:38:22,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:22,268 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:38:23,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:27,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:27,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:38:27,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:38:33,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:33,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:36,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:38:38,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:42,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:38:42,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:38:43,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:43,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:48,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:51,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:38:52,268 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:38:53,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:38:54,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:38:57,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:38:57,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:38:59,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:04,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:05,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:06,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:39:11,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:11,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:39:11,937 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:39:11,937 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:39:11,939 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:39:12,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:39:12,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:39:12,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:39:13,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:14,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:16,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:21,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:39:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:22,268 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:39:24,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:27,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:27,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:39:27,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:39:32,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:34,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:38,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:42,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:39:42,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:39:43,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:44,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:48,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:52,269 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:39:53,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:39:54,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:39:57,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:39:57,529 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:39:59,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:03,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:05,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:10,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:11,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:12,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:40:12,530 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:40:16,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:21,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:22,270 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:40:27,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:27,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:40:27,532 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:40:31,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:32,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:38,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:41,842 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:42,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:40:42,530 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:40:43,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:48,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:51,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:40:52,270 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:40:54,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:40:58,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:40:58,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:41:00,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:02,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:05,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:11,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:12,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:13,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:41:13,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:41:16,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:22,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:22,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:22,271 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:41:27,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:28,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:41:28,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:41:32,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:32,404 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:37,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:37,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:41:40,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:42,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:43,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:41:43,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:41:48,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:50,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:41:52,271 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:41:52,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:41:53,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:58,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:41:58,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:41:58,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:42:00,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:04,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:07,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:09,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:14,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:15,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:42:15,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:42:18,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:20,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:22,272 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:42:22,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:42:25,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:25,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:42:25,996 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:42:25,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:42:25,998 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:42:26,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:42:27,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:28,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:30,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:42:30,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:42:31,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:36,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:38,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:42:42,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:45,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:42:45,356 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:42:48,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:52,272 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:42:53,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:42:58,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:00,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:43:00,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:43:02,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:04,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:09,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:12,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:15,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:15,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:43:15,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:43:20,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:21,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:22,278 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:43:25,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:30,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:43:30,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:43:30,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:31,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:35,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:38,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:43:41,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:41,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:45,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:43:45,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:43:46,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:49,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:43:52,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:43:52,273 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:43:53,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:43:57,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:00,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:00,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:44:00,349 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:44:03,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:08,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:08,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:44:10,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:13,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:16,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:44:16,472 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:44:19,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:20,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:22,274 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:44:23,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:44:24,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:28,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:30,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:31,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:44:31,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:44:35,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:38,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:38,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:44:40,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:45,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:46,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:44:46,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:44:48,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:44:51,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:52,274 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:44:54,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:44:56,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:44:58,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:01,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:45:01,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:45:02,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:08,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:08,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:09,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:45:13,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:16,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:45:16,472 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:45:16,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:18,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:22,276 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:45:24,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:24,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:45:26,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:29,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:32,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:45:32,674 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:45:34,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:36,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:39,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:45:40,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:44,778 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:46,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:47,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:45:47,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:45:51,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:45:52,277 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:45:53,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:45:53,381 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:45:53,381 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:45:53,384 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:45:53,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:45:54,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:45:54,843 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:55,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:45:57,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:02,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:46:02,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:46:02,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:04,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:07,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:10,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:46:13,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:13,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:17,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:46:17,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:46:19,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:22,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:22,277 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:46:25,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:30,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:32,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:32,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:46:32,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:46:35,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:41,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:42,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:46,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:48,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:46:48,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:46:50,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:46:52,278 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:46:52,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:46:57,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:46:58,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:00,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:03,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:47:03,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:47:03,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:08,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:10,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:12,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:47:14,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:18,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:47:18,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:47:18,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:19,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:22,279 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:47:25,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:27,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:47:28,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:30,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:33,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:47:33,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:47:36,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:39,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:41,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:42,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:47:47,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:48,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:47:48,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:47:48,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:52,279 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:47:53,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:47:57,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:47:58,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:47:59,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:03,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:48:03,673 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:48:04,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:06,257 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:09,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:12,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:48:15,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:16,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:18,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:48:18,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:48:20,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:22,293 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:48:24,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:26,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:31,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:33,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:48:33,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:48:34,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:36,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:41,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:44,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:46,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:48,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:48:48,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:48:51,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:48:52,281 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:48:54,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:48:57,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:01,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:49:01,226 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:49:01,226 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:49:01,228 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:49:01,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:49:02,658 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:03,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:03,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:03,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:49:03,679 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:49:08,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:12,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:14,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:18,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:49:18,879 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:49:20,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:22,281 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:49:22,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:25,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:28,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:49:30,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:30,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:33,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:49:33,878 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:49:36,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:40,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:41,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:43,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:49:47,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:48,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:49:48,878 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:49:48,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:49:52,282 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:49:53,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:58,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:49:58,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:49:59,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:03,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:03,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:50:03,878 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:50:08,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:09,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:13,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:50:14,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:18,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:50:18,878 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:50:19,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:20,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:22,283 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:50:25,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:28,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:50:29,236 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:30,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:33,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:50:33,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:50:35,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:37,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:40,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:43,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:50:46,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:46,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:48,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:50:48,938 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:50:52,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:52,285 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:50:56,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:50:57,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:50:58,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:51:02,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:03,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:51:03,938 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:51:04,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:08,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:13,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:13,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:51:14,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:18,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:18,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:51:18,939 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:51:22,284 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:51:23,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:24,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:28,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:51:29,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:33,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:51:33,939 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:51:34,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:35,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:40,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:42,940 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:43,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:51:45,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:48,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:51:48,941 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:51:51,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:52,285 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:51:53,007 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:51:56,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:51:59,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:52:01,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:03,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:03,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:52:03,939 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:52:07,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:10,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:52:10,348 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:52:10,349 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:52:10,351 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:52:11,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:52:12,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:12,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:13,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:17,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:18,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:52:18,940 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:52:22,287 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:52:23,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:27,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:28,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:33,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:33,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:52:33,941 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:52:35,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:39,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:44,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:45,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:48,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:52:48,940 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:52:50,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:52:52,288 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:52:55,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:52:55,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:00,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:53:01,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:02,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:03,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:53:03,940 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:53:07,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:12,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:12,588 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:17,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:19,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:53:19,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:53:22,288 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:53:22,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:23,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:28,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:32,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:33,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:34,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:53:34,870 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:53:39,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:40,787 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:44,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:45,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:53:49,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:53:49,870 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:53:50,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:53:50,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:53:52,289 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:53:55,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:00,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:00,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:04,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:54:04,870 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:54:06,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:10,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:11,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:16,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:54:17,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:19,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:19,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:54:19,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:54:22,289 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:54:22,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:28,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:29,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:31,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:54:33,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:35,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:54:35,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:54:38,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:39,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:43,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:46,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:54:49,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:54:49,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:50,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:54:50,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:54:52,289 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:54:55,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:54:57,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:00,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:01,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:55:05,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:55:05,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:55:06,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:07,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:11,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:16,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:55:16,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:17,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:20,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:55:20,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:55:22,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:22,291 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:55:23,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:55:23,674 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:55:23,674 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:55:23,676 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:55:24,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:55:25,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:26,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:27,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:32,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:32,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:55:35,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:35,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:55:35,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:55:38,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:43,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:44,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:47,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:55:48,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:51,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:55:51,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:55:52,291 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:55:52,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:55:54,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:55:59,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:02,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:56:02,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:04,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:06,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:56:06,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:56:10,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:12,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:15,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:18,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:56:21,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:21,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:56:21,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:56:21,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:22,295 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:56:27,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:31,083 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:32,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:36,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:56:36,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:56:38,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:39,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:43,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:49,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:56:49,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:56:51,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:56:51,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:56:52,292 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:56:55,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:00,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:05,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:06,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:57:06,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:57:11,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:16,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:21,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:22,296 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:57:23,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:57:23,391 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:57:27,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:57:27,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:32,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:34,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:57:37,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:57:38,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:38,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:57:38,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:57:43,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:47,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:57:49,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:49,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:57:52,294 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:57:54,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:57:54,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:57:54,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:57:57,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:57:59,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:04,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:58:05,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:06,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:09,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:58:09,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:58:10,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:16,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:16,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:21,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:22,296 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:58:24,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:58:24,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:58:27,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:27,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:32,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:37,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:37,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:39,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:58:39,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:58:42,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:45,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:48,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:52,297 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:58:54,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:58:54,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:58:54,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:58:55,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:58:59,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:03,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 21:59:03,843 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 21:59:03,844 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 21:59:03,846 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 21:59:04,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 21:59:04,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:04,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:05,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:09,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:59:09,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:59:09,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:15,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:15,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:19,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 21:59:20,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:22,297 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:59:24,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:59:24,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:59:25,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:25,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:30,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:33,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:36,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:40,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:59:40,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:59:42,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:43,681 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:47,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:51,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 21:59:52,298 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 21:59:52,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 21:59:55,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 21:59:55,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 21:59:57,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:01,817 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:02,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:07,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:10,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:00:10,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:00:11,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:13,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:18,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:19,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:22,298 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:00:23,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:25,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:00:25,062 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:00:29,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:30,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:34,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:39,177 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:39,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:40,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:00:40,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:00:45,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:49,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:00:50,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:52,299 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:00:55,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:00:55,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:00:56,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:00:59,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:01,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:06,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:09,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:10,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:01:10,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:01:12,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:17,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:19,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:22,299 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:01:23,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:25,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:01:25,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:01:27,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:28,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:33,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:35,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:01:37,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:39,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:40,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:01:40,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:01:44,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:47,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:01:50,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:01:50,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:52,314 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:01:55,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:01:55,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:01:55,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:01:57,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:00,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:05,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:02:06,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:06,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:10,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:02:10,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:02:11,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:13,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:02:13,798 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:02:13,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:02:13,802 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:02:14,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:02:16,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:17,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:17,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:20,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:02:22,303 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:02:22,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:25,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:02:25,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:02:26,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:27,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:32,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:35,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:02:36,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:38,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:42,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:02:42,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:02:44,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:45,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:49,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:51,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:02:52,302 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:02:53,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:02:54,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:02:57,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:02:57,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:03:00,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:03,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:05,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:06,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:03:10,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:11,474 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:12,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:03:12,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:03:16,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:21,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:03:21,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:21,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:22,303 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:03:27,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:03:27,038 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:03:27,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:31,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:32,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:36,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:03:38,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:41,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:42,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:03:42,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:03:44,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:49,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:51,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:03:51,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:03:52,303 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:03:54,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:03:58,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:03:58,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:03:59,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:01,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:04,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:07,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:04:10,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:11,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:13,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:04:13,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:04:15,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:19,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:20,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:22,304 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:04:25,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:28,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:04:28,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:04:30,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:31,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:36,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:40,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:41,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:43,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:04:43,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:04:47,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:50,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:04:52,304 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:04:52,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:04:58,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:04:58,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:04:58,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:00,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:03,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:08,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:08,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:13,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:05:13,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:05:14,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:18,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:19,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:22,306 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:05:23,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:05:24,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:05:24,776 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:05:24,778 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:05:24,780 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:05:25,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:05:25,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:26,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:28,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:05:28,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:05:31,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:35,548 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:36,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:38,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:05:42,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:43,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:05:43,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:05:45,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:48,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:52,308 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:05:53,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:05:53,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:05:55,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:05:58,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:05:58,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:05:59,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:03,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:06:05,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:08,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:06:10,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:13,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:06:13,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:06:13,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:06:16,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:21,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:22,308 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:06:23,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:06:23,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:06:26,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:28,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:06:28,170 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:06:31,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:33,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:06:36,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:38,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:06:42,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:43,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:06:43,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:06:48,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:52,308 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:06:53,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:06:53,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:06:54,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:06:58,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:06:58,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:06:59,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:04,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:04,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:08,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:07:10,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:13,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:07:13,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:07:16,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:18,405 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:21,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:22,314 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:07:24,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:07:26,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:28,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:07:28,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:07:28,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:32,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:37,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:38,549 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:43,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:43,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:07:43,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:07:46,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:48,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:52,311 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:07:54,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:07:54,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:07:55,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:07:58,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:07:58,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:07:59,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:05,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:05,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:08:09,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:08:11,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:13,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:08:13,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:08:13,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:08:16,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:21,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:22,311 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:08:27,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:28,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:08:28,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:08:32,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:33,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:08:37,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:39,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:08:43,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:43,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:08:43,551 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:08:43,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:08:48,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:52,311 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:08:54,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:08:54,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:08:54,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:59,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:08:59,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:08:59,704 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:09:00,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:09:00,761 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:09:00,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:09:00,764 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:09:01,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:09:02,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:03,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:04,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:09,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:09:09,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:12,209 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:14,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:09:14,703 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:09:14,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:19,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:20,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:22,312 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:09:24,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:09:25,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:29,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:09:29,703 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:09:30,341 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:30,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:35,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:39,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:09:40,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:41,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:46,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:09:46,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:09:46,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:48,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:09:51,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:52,312 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:09:55,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:09:56,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:09:58,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:01,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:10:01,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:10:02,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:08,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:08,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:13,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:16,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:10:16,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:10:16,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:19,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:22,315 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:10:24,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:26,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:30,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:31,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:10:31,208 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:10:35,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:36,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:41,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:45,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:46,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:10:46,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:10:47,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:52,316 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:10:53,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:10:53,994 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:10:58,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:01,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:11:01,666 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:11:03,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:04,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:08,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:10,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:11:14,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:14,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:16,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:11:16,664 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:11:19,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:22,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:22,316 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:11:25,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:25,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:11:30,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:31,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:11:31,664 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:11:32,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:35,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:40,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:11:41,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:42,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:46,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:47,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:11:47,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:11:50,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:11:52,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:52,317 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:11:57,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:11:57,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:12:00,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:12:02,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:02,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:12:02,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:12:08,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:08,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:12:08,163 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:12:08,164 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:12:08,178 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:12:08,499 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:12:09,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:12:10,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:12:12,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:12:13,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:17,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:12:17,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:12:19,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:22,317 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:12:24,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:27,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:12:29,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:32,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:12:32,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:12:35,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:40,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:42,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:12:45,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:47,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:12:47,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:12:51,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:52,318 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:12:56,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:12:57,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:13:01,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:13:02,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:13:07,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:12,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:12,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:13:17,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:17,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:13:17,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:13:22,318 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:13:23,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:27,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:13:28,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:32,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:13:32,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:13:34,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:39,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:42,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:13:44,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:47,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:13:47,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:13:50,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:52,318 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:13:55,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:13:57,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:14:00,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:14:02,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:14:06,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:11,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:12,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:14:16,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:17,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:14:17,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:14:22,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:22,319 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:14:27,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:27,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:14:32,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:32,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:14:32,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:14:38,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:42,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:14:43,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:47,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:14:47,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:14:49,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:52,319 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:14:54,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:14:57,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:14:59,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:15:02,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:15:05,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:10,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:12,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:15:15,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:17,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:15:17,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:15:21,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:22,320 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:15:26,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:27,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:15:31,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:32,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:15:32,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:15:37,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:42,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:42,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:15:47,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:47,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:15:47,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:15:52,320 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:15:53,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:15:57,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:15:58,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:02,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:16:02,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:16:04,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:09,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:12,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:16:14,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:17,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:16:17,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:16:20,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:22,321 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:16:25,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:27,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:16:30,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:32,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:16:32,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:16:36,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:41,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:42,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:16:46,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:47,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:16:47,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:16:52,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:52,321 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:16:57,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:16:57,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:17:02,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:02,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:17:02,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:17:08,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:12,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:17:13,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:17,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:17:17,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:17:19,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:22,321 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:17:24,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:27,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:17:29,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:32,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:17:32,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:17:35,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:40,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:42,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:17:45,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:47,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:17:47,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:17:51,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:52,336 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:17:56,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:17:57,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:18:01,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:02,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:18:02,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:18:07,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:12,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:12,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:18:17,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:17,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:18:17,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:18:22,322 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:18:23,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:28,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:32,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:18:32,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:18:34,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:39,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:44,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:47,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:18:47,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:18:50,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:18:52,323 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:18:55,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:00,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:02,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:19:02,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:19:06,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:11,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:16,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:17,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:19:17,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:19:22,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:22,323 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:19:27,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:32,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:32,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:19:32,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:19:38,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:43,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:47,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:19:47,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:19:49,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:52,324 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:19:54,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:19:59,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:02,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:20:02,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:20:05,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:10,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:15,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:17,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:20:17,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:20:21,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:22,324 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:20:26,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:31,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:32,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:20:32,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:20:37,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:42,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:47,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:47,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:20:47,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:20:52,325 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:20:53,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:20:58,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:02,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:21:02,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:21:04,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:09,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:14,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:17,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:21:17,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:21:20,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:22,325 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:21:25,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:30,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:32,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:21:32,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:21:36,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:41,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:46,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:47,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:21:47,959 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:21:52,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:21:52,325 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:21:57,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:02,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:02,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:22:02,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:22:08,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:13,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:17,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:22:17,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:22:19,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:21,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:22:21,507 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:22:21,507 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:22:21,510 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:22:22,326 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:22:22,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:22:23,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:22:24,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:29,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:32,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:22:32,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:22:35,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:40,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:45,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:45,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:22:48,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:22:48,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:22:50,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:52,326 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:22:56,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:22:57,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:23:01,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:03,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:23:03,095 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:23:07,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:12,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:13,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:23:15,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:23:17,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:22,336 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:23:23,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:28,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:29,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:23:33,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:35,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:23:35,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:23:38,016 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:23:39,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:44,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:48,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:23:49,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:52,328 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:23:54,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:23:59,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:01,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:24:04,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:09,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:14,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:19,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:22,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:24:22,328 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:24:25,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:30,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:35,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:24:35,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:24:35,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:40,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:45,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:50,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:24:50,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:24:50,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:24:52,332 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:24:56,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:01,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:05,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:25:05,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:25:06,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:11,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:16,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:20,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:25:20,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:25:21,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:22,331 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:25:27,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:29,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:25:32,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:35,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:25:35,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:25:37,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:42,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:45,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:25:48,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:50,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:25:50,164 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:25:52,333 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:25:53,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:25:55,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:25:58,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:03,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:05,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:26:05,160 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:26:07,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:26:09,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:14,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:17,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:26:19,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:20,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:26:20,160 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:26:22,333 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:26:25,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:27,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:26:30,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:35,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:36,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:26:36,646 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:26:39,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:26:41,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:46,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:51,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:26:51,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:26:51,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:26:51,644 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:26:52,334 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:26:57,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:01,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:27:02,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:06,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:27:06,647 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:27:07,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:11,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:27:12,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:17,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:21,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:27:21,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:27:22,335 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:27:23,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:23,722 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:27:28,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:33,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:33,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:27:36,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:27:36,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:27:38,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:43,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:27:44,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:49,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:51,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:27:51,779 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:27:52,337 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:27:55,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:27:55,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:00,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:05,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:05,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:06,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:28:06,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:28:10,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:15,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:16,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:21,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:21,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:28:21,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:28:22,341 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:28:27,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:27,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:32,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:36,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:28:36,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:28:37,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:38,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:43,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:48,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:49,641 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:28:51,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:28:51,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:28:52,338 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:28:54,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:28:59,733 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:00,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:05,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:07,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:29:07,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:29:08,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:29:08,602 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:29:08,604 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:29:08,607 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:29:08,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:29:09,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:10,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:10,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:15,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:19,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:21,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:22,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:29:22,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:29:22,348 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:29:26,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:31,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:31,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:36,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:37,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:29:37,125 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:29:42,089 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:42,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:47,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:52,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:29:52,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:29:52,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:29:52,421 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:29:53,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:29:58,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:03,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:04,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:07,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:30:07,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:30:09,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:14,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:14,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:19,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:22,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:30:22,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:30:22,340 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:30:23,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:25,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:30,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:33,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:35,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:37,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:30:37,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:30:41,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:43,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:46,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:51,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:30:52,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:30:52,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:30:52,410 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:30:55,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:30:57,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:02,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:05,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:31:07,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:31:07,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:31:07,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:12,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:15,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:31:18,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:22,343 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:31:22,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:31:22,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:31:24,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:25,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:31:29,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:34,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:36,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:31:37,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:31:37,826 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:31:40,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:45,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:46,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:31:50,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:52,345 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:31:53,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:31:53,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:31:55,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:31:56,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:00,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:05,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:06,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:08,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:32:08,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:32:11,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:16,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:16,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:21,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:22,346 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:32:23,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:32:23,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:32:26,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:27,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:32,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:33,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:32:33,878 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:32:33,879 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:32:33,881 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:32:34,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:32:34,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:35,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:37,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:38,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:32:38,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:32:43,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:45,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:48,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:52,347 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:32:53,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:32:53,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:32:53,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:32:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:32:59,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:03,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:05,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:08,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:33:08,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:33:10,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:13,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:15,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:20,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:22,348 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:33:23,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:33:23,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:33:23,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:26,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:31,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:34,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:36,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:38,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:33:38,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:33:42,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:44,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:47,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:52,349 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:33:52,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:33:53,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:33:53,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:33:54,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:33:58,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:04,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:04,248 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:08,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:34:08,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:34:09,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:14,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:15,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:20,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:22,349 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:34:23,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:34:23,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:34:24,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:25,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:30,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:32,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:36,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:39,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:34:39,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:34:42,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:42,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:47,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:52,350 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:34:52,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:34:52,551 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:34:54,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:34:54,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:34:57,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:02,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:03,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:08,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:09,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:35:09,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:35:12,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:13,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:18,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:20,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:22,350 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:35:23,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:24,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:35:24,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:35:29,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:34,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:37,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:39,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:39,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:35:39,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:35:45,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:47,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:50,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:52,351 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:35:54,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:35:54,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:35:56,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:35:57,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:35:57,563 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:35:57,563 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:35:57,565 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:35:58,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:35:58,010 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:35:59,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:01,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:06,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:08,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:09,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:36:09,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:36:11,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:17,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:18,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:22,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:22,351 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:36:24,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:36:24,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:36:28,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:28,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:33,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:38,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:39,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:39,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:36:39,988 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:36:44,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:48,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:36:49,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:52,353 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:36:54,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:36:54,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:36:55,000 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:36:56,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:00,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:05,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:06,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:09,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:37:09,994 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:37:11,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:16,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:16,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:21,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:22,354 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:37:24,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:24,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:37:24,995 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:37:27,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:32,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:34,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:37,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:41,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:37:41,022 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:37:43,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:44,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:48,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:52,354 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:37:52,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:37:53,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:37:56,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:37:56,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:37:59,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:02,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:04,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:09,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:11,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:38:11,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:38:12,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:15,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:21,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:22,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:22,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:38:26,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:38:26,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:38:26,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:30,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:31,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:36,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:40,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:41,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:38:41,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:38:42,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:47,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:50,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:38:52,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:38:52,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:56,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:38:56,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:38:57,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:38:58,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:02,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:06,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:39:06,675 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:39:06,676 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:39:06,678 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:39:07,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:39:07,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:08,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:09,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:11,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:39:11,474 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:39:12,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:17,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:18,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:22,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:39:23,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:26,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:39:26,475 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:39:28,530 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:28,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:33,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:38,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:39,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:41,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:39:41,474 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:39:44,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:46,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:50,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:52,357 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:39:55,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:39:56,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:39:57,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:39:57,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:40:00,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:05,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:08,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:11,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:12,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:40:12,564 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:40:16,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:16,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:21,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:22,359 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:40:26,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:27,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:27,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:40:27,564 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:40:32,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:37,059 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:38,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:42,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:40:42,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:40:43,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:45,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:48,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:52,360 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:40:54,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:40:54,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:40:57,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:40:57,564 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:40:59,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:04,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:05,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:10,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:12,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:12,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:41:12,565 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:41:15,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:20,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:22,360 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:41:22,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:26,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:27,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:41:27,565 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:41:31,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:32,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:36,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:40,497 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:42,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:41:42,565 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:41:42,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:47,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:41:47,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:50,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:41:52,370 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:41:53,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:41:57,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:41:57,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:41:58,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:00,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:02,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:42:03,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:09,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:10,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:12,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:42:12,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:42:14,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:17,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:42:17,393 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:42:17,395 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:42:17,397 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:42:17,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:42:17,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:42:18,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:20,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:22,361 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:42:26,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:27,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:42:27,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:42:28,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:31,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:32,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:42:37,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:38,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:42,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:42,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:42:42,568 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:42:46,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:47,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:47,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:42:52,363 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:42:53,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:42:57,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:42:57,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:42:57,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:42:58,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:03,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:43:03,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:05,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:09,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:13,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:43:13,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:43:14,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:17,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:18,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:43:19,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:22,364 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:43:24,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:25,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:28,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:43:28,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:43:30,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:33,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:43:35,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:35,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:40,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:43,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:43:43,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:43:43,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:46,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:48,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:43:51,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:52,364 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:43:53,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:43:56,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:43:58,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:43:58,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:44:02,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:02,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:03,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:44:07,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:12,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:12,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:13,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:44:13,058 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:44:18,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:18,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:44:20,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:22,375 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:44:23,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:28,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:44:28,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:44:29,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:30,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:33,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:44:34,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:39,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:40,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:43,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:44:43,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:44:45,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:48,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:44:48,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:44:50,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:52,365 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:44:56,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:44:58,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:44:58,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:44:58,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:02,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:03,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:45:07,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:09,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:12,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:13,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:45:13,062 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:45:17,115 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:18,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:18,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:45:22,367 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:45:23,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:25,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:45:25,031 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:45:25,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:45:25,034 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:45:25,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:45:27,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:28,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:45:28,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:45:29,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:33,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:45:34,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:35,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:39,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:43,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:45:43,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:45:44,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:45,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:48,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:45:49,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:52,369 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:45:55,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:45:55,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:45:58,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:45:58,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:46:01,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:03,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:46:05,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:06,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:11,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:13,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:46:13,060 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:46:13,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:17,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:19,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:46:22,368 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:46:22,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:23,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:27,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:28,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:46:28,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:46:31,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:32,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:34,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:46:37,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:40,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:43,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:43,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:46:43,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:46:49,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:49,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:46:50,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:52,369 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:46:54,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:46:58,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:46:58,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:46:58,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:47:00,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:05,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:06,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:47:08,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:10,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:13,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:47:13,876 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:47:16,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:18,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:21,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:47:21,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:22,369 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:47:26,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:27,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:28,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:47:28,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:47:33,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:36,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:47:37,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:38,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:43,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:43,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:47:44,001 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:47:47,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:49,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:51,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:47:52,371 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:47:55,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:47:55,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:47:58,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:47:58,999 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:48:00,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:05,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:05,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:06,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:48:10,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:13,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:48:13,999 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:48:15,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:16,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:21,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:21,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:48:22,382 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:48:23,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:26,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:30,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:48:30,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:48:31,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:31,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:48:31,656 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:48:31,657 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:48:31,658 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:48:32,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:48:33,558 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:36,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:48:36,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:41,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:41,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:45,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:48:45,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:48:47,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:51,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:48:51,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:48:52,371 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:48:53,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:48:58,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:00,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:49:00,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:49:01,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:03,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:07,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:49:08,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:11,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:13,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:15,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:49:15,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:49:19,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:22,373 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:49:22,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:49:24,902 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:25,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:30,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:31,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:49:31,818 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:49:34,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:35,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:37,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:49:40,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:43,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:46,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:46,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:49:46,815 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:49:52,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:49:52,373 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:49:52,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:49:53,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:49:57,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:01,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:01,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:50:01,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:50:03,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:08,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:08,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:50:11,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:13,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:16,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:50:16,817 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:50:18,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:21,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:22,374 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:50:23,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:50:24,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:29,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:31,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:50:31,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:50:35,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:38,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:50:39,438 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:40,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:45,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:46,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:50:46,817 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:50:49,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:50:51,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:52,374 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:50:53,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:50:56,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:50:59,581 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:01,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:01,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:51:01,817 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:51:06,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:07,637 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:08,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:51:11,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:16,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:16,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:51:16,817 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:51:17,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:21,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:22,376 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:51:23,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:51:25,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:27,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:32,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:33,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:51:33,244 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:51:35,842 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:38,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:38,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:51:43,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:44,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:51:44,100 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:51:44,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:51:44,102 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:51:44,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:51:45,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:51:48,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:51:48,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:51:49,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:52,377 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:51:53,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:51:55,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:51:55,982 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:00,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:03,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:52:03,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:52:03,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:05,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:10,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:52:10,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:13,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:15,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:19,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:52:19,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:52:20,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:21,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:22,377 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:52:25,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:52:26,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:31,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:32,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:34,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:52:34,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:52:37,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:40,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:52:41,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:43,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:48,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:49,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:52:49,395 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:52:51,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:52:52,377 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:52:53,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:55,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:52:59,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:52:59,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:04,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:04,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:53:04,398 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:53:09,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:10,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:10,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:53:15,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:17,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:19,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:53:19,396 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:53:20,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:22,380 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:53:25,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:53:26,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:27,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:31,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:34,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:53:34,398 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:53:37,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:37,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:40,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:53:42,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:46,031 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:47,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:49,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:53:49,396 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:53:52,381 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:53:52,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:53:55,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:53:56,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:53:57,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:03,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:04,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:04,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:54:04,396 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:54:09,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:10,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:54:14,220 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:14,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:19,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:54:19,734 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:54:19,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:22,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:22,381 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:54:25,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:25,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:54:30,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:33,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:34,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:54:34,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:54:35,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:40,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:54:40,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:41,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:45,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:49,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:54:49,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:54:50,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:54:50,596 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:54:50,597 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:54:50,599 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:54:51,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:54:51,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:51,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:54:52,381 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:54:52,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:54:55,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:54:57,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:01,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:02,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:04,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:55:04,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:55:07,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:09,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:10,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:55:13,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:19,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:19,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:55:19,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:55:19,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:22,382 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:55:25,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:25,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:55:27,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:31,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:35,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:55:35,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:55:36,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:37,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:40,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:55:41,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:46,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:47,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:50,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:55:50,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:55:52,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:55:52,384 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:55:55,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:55:58,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:55:58,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:03,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:05,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:56:05,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:56:06,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:08,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:10,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:56:13,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:16,125 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:18,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:20,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:56:20,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:56:22,385 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:56:24,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:24,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:25,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:56:29,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:34,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:35,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:35,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:56:35,145 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:56:40,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:40,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:56:44,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:46,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:50,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:56:50,399 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:56:51,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:56:52,385 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:56:54,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:56:55,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:56:57,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:02,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:04,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:05,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:57:05,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:57:07,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:10,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:57:11,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:12,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:17,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:20,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:57:20,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:57:21,698 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:22,385 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:57:23,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:25,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:57:28,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:31,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:33,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:35,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:57:35,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:57:39,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:40,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:57:41,882 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:44,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:49,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:49,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:57:50,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:57:50,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:57:52,386 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:57:55,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:57:55,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:57:58,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 22:57:58,876 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 22:57:58,876 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 22:57:58,879 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 22:57:59,031 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 22:58:00,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:00,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:05,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:06,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:58:06,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:58:10,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:11,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:16,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:20,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:21,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:58:21,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:58:21,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:22,386 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:58:27,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:28,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:32,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:36,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:58:36,805 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:58:38,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:38,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:43,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:48,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:48,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:51,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:58:51,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:58:52,388 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:58:54,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:58:56,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:58:59,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:04,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:06,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:06,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:59:06,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:59:10,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:11,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:59:14,589 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:15,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:20,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:21,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:59:21,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:59:22,388 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:59:24,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:25,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:26,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:59:30,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:33,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:35,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:36,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:59:36,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:59:40,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:41,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:41,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 22:59:45,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:51,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:51,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 22:59:51,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 22:59:51,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 22:59:52,388 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 22:59:56,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 22:59:58,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:00:01,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:00:02,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:06,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:00:06,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:00:08,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:11,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:00:13,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:00:13,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:18,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:21,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:00:21,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:00:22,389 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:00:24,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:28,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:00:28,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:00:29,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:34,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:36,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:00:36,808 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:00:40,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:00:40,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:43,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:00:45,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:50,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:00:50,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:51,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:00:51,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:00:52,389 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:00:56,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:00:58,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:01:00,279 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:02,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:07,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:08,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:01:08,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:01:12,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:12,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:13,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:01:17,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:19,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:01:19,605 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:01:19,606 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:01:19,608 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:01:20,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:01:20,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:22,400 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:01:22,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:23,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:01:23,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:01:23,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:28,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:01:28,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:30,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:34,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:38,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:01:38,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:01:39,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:40,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:43,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:01:44,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:50,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:01:50,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:52,390 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:01:53,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:01:53,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:01:56,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:01:58,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:02:00,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:01,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:06,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:08,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:02:08,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:02:10,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:13,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:02:17,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:18,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:22,391 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:02:22,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:23,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:02:23,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:02:27,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:28,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:02:28,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:32,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:38,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:38,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:02:38,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:02:38,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:43,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:02:43,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:48,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:49,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:52,391 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:02:53,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:02:53,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:02:54,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:02:58,105 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:02:58,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:03:00,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:05,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:08,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:03:08,119 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:03:08,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:03:10,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:14,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:03:16,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:16,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:03:21,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:22,393 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:03:23,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:03:23,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:03:26,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:03:26,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:29,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:03:31,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:36,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:38,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:03:38,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:03:40,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:03:42,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:44,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:03:47,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:50,442 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:03:52,393 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:03:52,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:53,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:03:53,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:03:58,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:03:59,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:04:00,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:04,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:08,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:04:08,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:04:09,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:10,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:14,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:04:14,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:20,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:20,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:22,393 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:04:23,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:04:23,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:04:25,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:29,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:04:30,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:30,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:35,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:37,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:04:37,120 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:04:37,120 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:04:37,122 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:04:37,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:04:38,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:04:38,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:04:38,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:41,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:44,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:04:46,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:48,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:04:52,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:52,394 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:04:53,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:04:53,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:04:57,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:04:59,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:04:59,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:02,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:07,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:08,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:05:08,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:05:09,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:13,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:14,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:05:16,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:18,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:22,395 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:05:24,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:05:24,379 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:05:24,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:26,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:29,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:05:29,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:34,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:36,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:39,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:05:39,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:05:40,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:44,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:05:46,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:46,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:51,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:52,397 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:05:54,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:05:54,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:05:56,466 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:05:56,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:05:59,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:06:01,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:04,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:07,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:09,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:06:09,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:06:12,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:14,581 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:14,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:06:18,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:22,397 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:06:23,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:24,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:06:24,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:06:24,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:28,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:31,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:06:34,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:34,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:39,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:39,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:06:39,396 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:06:44,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:44,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:46,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:06:49,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:52,397 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:06:54,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:06:54,393 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:06:54,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:06:54,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:06:59,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:01,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:07:02,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:05,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:09,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:07:09,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:07:11,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:13,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:16,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:07:16,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:21,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:22,398 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:07:23,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:24,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:07:24,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:07:27,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:31,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:07:33,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:33,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:38,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:39,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:07:39,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:07:41,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:43,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:46,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:07:48,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:49,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:07:49,479 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:07:49,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:07:49,482 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:07:50,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:07:51,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:07:52,409 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:07:53,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:07:54,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:07:54,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:07:59,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:01,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:08:01,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:04,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:09,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:09,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:08:09,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:08:11,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:15,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:16,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:08:18,470 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:20,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:22,399 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:08:25,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:25,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:08:25,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:08:28,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:31,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:31,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:08:36,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:38,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:40,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:08:40,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:08:41,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:46,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:08:47,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:48,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:08:52,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:52,399 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:08:55,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:08:55,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:08:57,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:08:58,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:01,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:09:02,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:06,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:07,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:10,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:09:10,667 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:09:13,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:16,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:09:16,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:19,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:22,401 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:09:24,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:25,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:09:25,667 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:09:27,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:30,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:31,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:09:36,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:37,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:40,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:09:40,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:09:41,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:46,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:09:47,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:47,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:52,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:09:52,402 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:09:55,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:09:55,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:09:55,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:09:57,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:01,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:10:02,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:05,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:08,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:11,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:10:11,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:10:13,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:15,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:16,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:10:18,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:22,403 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:10:23,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:23,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:26,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:10:26,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:10:28,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:31,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:10:33,521 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:33,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:38,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:41,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:10:41,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:10:41,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:44,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:48,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:10:49,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:50,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:10:52,403 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:10:55,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:10:56,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:10:56,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:10:59,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:10:59,017 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:10:59,018 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:10:59,020 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:10:59,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:11:00,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:01,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:03,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:11:06,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:10,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:11,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:11,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:11:11,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:11:16,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:18,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:11:18,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:22,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:22,403 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:11:26,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:11:26,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:11:27,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:28,882 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:32,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:33,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:11:38,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:38,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:41,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:11:41,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:11:43,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:48,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:11:49,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:11:49,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:52,404 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:11:55,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:11:56,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:11:56,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:11:59,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:00,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:03,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:12:05,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:07,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:11,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:11,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:12:11,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:12:16,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:17,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:18,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:12:21,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:22,407 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:12:26,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:26,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:12:26,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:12:27,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:32,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:33,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:12:37,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:37,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:41,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:12:41,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:12:42,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:47,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:48,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:48,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:12:52,410 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:12:53,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:12:55,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:12:56,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:12:56,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:12:58,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:03,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:13:04,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:04,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:09,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:11,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:13:11,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:13:14,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:14,982 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:18,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:13:19,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:22,409 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:13:23,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:25,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:26,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:13:26,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:13:31,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:33,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:33,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:13:36,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:41,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:41,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:13:41,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:13:43,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:47,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:48,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:13:52,409 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:13:52,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:13:53,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:13:56,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:13:56,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:13:58,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:01,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:14:03,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:03,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:14:08,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:11,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:14:11,766 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:14:11,767 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:14:11,769 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:14:11,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:14:11,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:14:12,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:14:13,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:14:14,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:18,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:14:19,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:22,410 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:14:24,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:26,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:14:26,829 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:14:30,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:33,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:14:35,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:37,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:14:40,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:41,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:14:41,829 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:14:45,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:47,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:14:48,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:14:50,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:52,412 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:14:55,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:14:56,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:14:56,829 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:14:56,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:00,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:03,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:15:05,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:06,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:11,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:12,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:15:12,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:15:15,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:16,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:18,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:15:21,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:22,412 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:15:23,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:26,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:27,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:15:27,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:15:32,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:33,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:33,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:15:37,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:42,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:15:42,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:15:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:43,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:47,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:49,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:15:52,413 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:15:52,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:15:53,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:15:57,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:15:57,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:15:58,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:01,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:03,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:04,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:16:08,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:11,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:12,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:16:12,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:16:14,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:19,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:16:20,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:21,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:22,413 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:16:25,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:27,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:16:27,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:16:29,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:31,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:34,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:16:36,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:39,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:41,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:42,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:16:42,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:16:47,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:49,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:16:49,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:16:52,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:52,414 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:16:57,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:16:57,213 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:16:57,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:16:57,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:02,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:04,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:17:07,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:08,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:13,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:13,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:17:13,490 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:17:15,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:18,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:19,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:17:22,416 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:17:24,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:26,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:28,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:17:28,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:17:29,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:33,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:17:33,319 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:17:33,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:17:33,323 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:17:34,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:17:34,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:17:35,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:36,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:40,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:43,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:17:43,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:17:44,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:45,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:50,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:50,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:17:52,416 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:17:54,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:17:56,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:17:58,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:17:58,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:18:01,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:05,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:18:07,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:07,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:12,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:14,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:18:14,309 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:18:17,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:17,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:20,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:18:22,418 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:18:23,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:27,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:29,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:29,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:18:29,307 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:18:34,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:35,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:35,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:18:39,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:44,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:44,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:18:44,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:18:45,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:49,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:51,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:18:52,418 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:18:55,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:18:55,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:18:59,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:18:59,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:19:00,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:03,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:06,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:06,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:19:11,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:13,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:14,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:19:14,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:19:16,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:21,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:19:21,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:22,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:22,419 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:19:27,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:29,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:19:29,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:19:31,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:32,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:36,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:19:37,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:42,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:43,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:44,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:19:44,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:19:48,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:51,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:19:52,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:19:52,419 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:19:54,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:19:59,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:19:59,327 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:19:59,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:02,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:04,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:06,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:20:09,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:12,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:14,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:20:14,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:20:15,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:21,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:21,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:20:22,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:22,420 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:20:26,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:29,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:20:29,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:20:31,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:31,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:36,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:38,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:20:41,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:41,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:44,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:20:44,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:20:47,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:49,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:20:49,723 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:20:49,724 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:20:49,727 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:20:50,589 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:20:51,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:20:52,420 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:20:53,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:20:53,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:58,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:20:59,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:20:59,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:21:01,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:03,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:08,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:21:08,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:09,709 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:13,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:14,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:21:14,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:21:18,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:19,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:22,421 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:21:23,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:21:24,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:27,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:29,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:21:29,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:21:29,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:34,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:37,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:38,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:21:40,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:45,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:45,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:21:45,223 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:21:47,982 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:21:50,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:52,423 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:21:54,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:21:55,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:21:58,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:00,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:22:00,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:22:01,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:06,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:07,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:09,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:22:12,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:15,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:22:15,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:22:16,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:17,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:22,424 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:22:23,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:24,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:22:26,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:28,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:30,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:22:30,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:22:33,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:34,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:38,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:39,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:22:44,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:44,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:45,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:22:45,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:22:50,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:22:52,424 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:22:54,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:22:54,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:22:56,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:00,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:23:00,306 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:23:01,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:04,512 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:06,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:09,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:23:11,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:12,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:15,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:23:15,304 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:23:17,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:21,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:22,425 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:23:23,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:24,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:23:28,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:29,698 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:30,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:23:30,304 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:23:33,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:39,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:39,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:23:39,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:44,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:46,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:23:46,067 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:23:49,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:49,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:23:52,425 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:23:55,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:23:56,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:23:57,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:23:57,376 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:23:57,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:23:57,379 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:23:57,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:23:59,896 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:01,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:24:01,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:24:01,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:06,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:07,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:11,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:24:11,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:16,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:24:16,066 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:24:17,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:18,016 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:22,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:22,428 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:24:26,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:26,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:24:27,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:31,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:24:31,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:24:33,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:36,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:38,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:41,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:24:43,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:46,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:24:46,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:24:48,220 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:24:49,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:52,429 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:24:54,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:24:56,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:24:56,279 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:00,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:01,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:25:01,068 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:25:05,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:06,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:10,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:12,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:25:16,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:25:16,071 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:25:16,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:16,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:21,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:22,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:25:24,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:26,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:27,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:25:31,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:25:31,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:25:32,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:34,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:37,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:42,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:25:42,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:44,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:46,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:25:46,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:25:48,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:52,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:25:52,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:25:54,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:25:57,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:25:59,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:01,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:26:01,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:26:02,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:04,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:09,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:12,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:26:12,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:14,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:16,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:26:16,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:26:20,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:21,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:22,431 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:26:26,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:28,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:26:31,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:26:31,301 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:26:31,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:31,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:36,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:40,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:42,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:43,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:26:47,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:47,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:26:47,149 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:26:50,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:52,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:52,432 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:26:57,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:26:58,125 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:26:58,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:27:02,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:27:02,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:27:03,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:07,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:27:07,789 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:27:07,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:27:07,792 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:27:08,200 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:27:08,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:10,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:13,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:13,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:27:17,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:27:17,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:27:18,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:19,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:22,432 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:27:24,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:28,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:28,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:27:29,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:32,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:27:32,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:27:35,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:38,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:41,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:44,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:27:46,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:46,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:47,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:27:47,149 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:27:51,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:52,434 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:27:56,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:27:56,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:27:59,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:28:01,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:03,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:28:03,524 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:28:06,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:06,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:11,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:14,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:28:16,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:17,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:18,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:28:18,522 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:28:22,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:28:22,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:24,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:28,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:29,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:28:33,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:33,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:28:33,523 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:28:34,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:38,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:42,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:44,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:28:44,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:49,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:49,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:28:49,974 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:28:52,435 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:28:52,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:28:55,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:28:59,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:29:01,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:02,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:04,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:29:04,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:29:06,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:11,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:13,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:14,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:29:16,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:19,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:29:19,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:29:22,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:22,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:22,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:29:27,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:29,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:29:30,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:33,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:34,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:29:34,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:29:38,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:40,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:43,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:44,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:29:49,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:49,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:29:49,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:29:50,337 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:52,436 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:29:55,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:29:58,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:29:59,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:30:00,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:05,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:06,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:30:06,181 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:30:08,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:11,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:14,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:30:16,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:16,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:30:16,773 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:30:16,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:30:16,776 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:30:17,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:30:18,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:21,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:30:21,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:30:21,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:22,437 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:30:27,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:28,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:29,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:30:32,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:36,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:30:36,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:30:37,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:38,744 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:42,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:44,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:30:47,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:48,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:51,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:30:51,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:30:52,437 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:30:53,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:58,882 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:30:59,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:30:59,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:31:04,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:06,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:31:06,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:31:06,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:09,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:14,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:14,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:31:16,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:20,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:21,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:31:21,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:31:22,438 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:31:25,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:27,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:29,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:31:30,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:36,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:36,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:31:36,167 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:31:37,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:41,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:44,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:31:45,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:47,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:51,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:31:51,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:31:52,439 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:31:52,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:55,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:31:57,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:31:59,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:32:02,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:06,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:32:06,456 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:32:07,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:09,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:12,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:14,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:32:16,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:17,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:21,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:32:21,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:32:22,441 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:32:23,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:26,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:29,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:29,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:32:35,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:36,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:32:36,456 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:32:36,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:40,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:44,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:44,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:32:46,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:51,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:32:51,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:32:51,653 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:32:52,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:32:54,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:32:56,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:00,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:33:01,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:02,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:06,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:33:06,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:33:07,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:12,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:13,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:15,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:33:18,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:21,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:33:21,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:33:22,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:33:22,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:23,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:28,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:30,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:33:31,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:33:31,444 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:33:31,445 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:33:31,448 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:33:31,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:33:32,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:33,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:36,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:33:36,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:33:38,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:43,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:44,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:45,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:33:49,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:51,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:33:51,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:33:52,442 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:33:53,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:33:54,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:33:59,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:00,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:34:03,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:05,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:06,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:34:06,654 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:34:10,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:11,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:15,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:15,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:34:21,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:21,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:21,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:34:21,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:34:22,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:34:26,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:29,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:30,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:34:32,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:36,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:34:36,653 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:34:37,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:39,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:42,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:46,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:34:48,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:49,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:51,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:34:51,653 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:34:52,443 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:34:54,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:34:58,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:34:59,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:01,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:35:04,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:06,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:06,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:35:06,653 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:35:09,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:14,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:16,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:35:16,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:19,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:22,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:35:23,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:35:23,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:35:24,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:25,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:30,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:32,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:35:34,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:35,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:38,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:35:38,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:35:41,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:44,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:46,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:47,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:35:51,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:35:52,445 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:35:53,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:35:53,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:35:54,920 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:35:57,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:02,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:36:02,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:03,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:08,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:36:08,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:36:08,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:13,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:13,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:17,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:36:18,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:22,446 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:36:23,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:36:23,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:36:23,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:24,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:29,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:31,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:32,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:36:34,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:38,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:36:38,574 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:36:39,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:39,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:36:39,908 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:36:39,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:36:39,911 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:36:40,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:36:41,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:44,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:47,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:36:49,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:36:51,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:36:52,447 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:36:53,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:36:53,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:36:55,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:00,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:01,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:02,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:37:05,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:08,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:37:08,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:37:09,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:10,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:15,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:17,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:37:19,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:20,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:22,447 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:37:23,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:37:23,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:37:26,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:29,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:32,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:32,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:37:37,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:37,632 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:38,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:37:38,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:37:42,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:47,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:47,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:37:48,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:52,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:37:54,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:37:54,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:37:54,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:37:57,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:37:59,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:04,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:04,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:38:06,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:09,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:38:09,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:38:10,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:15,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:18,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:19,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:38:20,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:22,448 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:38:25,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:38:25,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:38:26,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:26,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:31,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:34,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:38:37,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:37,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:40,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:38:40,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:38:42,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:47,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:48,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:49,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:38:52,449 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:38:53,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:38:55,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:38:55,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:38:57,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:38:59,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:04,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:04,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:39:07,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:09,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:10,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:39:10,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:39:14,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:17,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:19,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:39:19,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:22,451 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:39:25,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:25,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:39:25,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:39:25,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:30,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:34,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:39:36,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:37,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:40,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:39:40,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:39:41,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:46,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:47,585 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:49,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:39:51,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:52,451 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:39:55,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:39:55,019 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:39:55,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:39:55,022 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:39:55,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:39:55,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:39:55,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:39:56,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:39:57,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:39:57,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:02,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:04,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:40:07,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:08,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:10,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:40:10,213 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:40:13,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:15,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:19,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:19,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:40:22,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:40:24,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:25,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:40:25,213 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:40:25,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:30,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:34,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:40:35,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:35,926 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:40,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:41,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:40:41,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:40:45,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:46,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:49,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:40:51,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:52,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:40:55,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:40:56,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:40:56,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:40:56,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:41:01,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:03,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:04,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:41:07,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:11,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:41:11,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:41:13,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:13,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:18,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:20,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:41:22,453 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:41:23,279 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:23,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:26,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:41:26,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:41:28,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:33,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:34,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:35,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:41:39,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:41,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:41:41,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:41:43,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:44,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:49,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:50,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:41:51,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:41:52,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:41:55,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:41:56,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:41:56,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:42:00,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:01,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:05,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:05,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:42:11,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:11,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:11,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:42:11,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:42:16,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:20,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:42:21,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:22,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:22,454 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:42:26,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:42:26,735 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:42:27,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:31,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:33,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:35,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:42:38,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:41,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:42:41,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:42:41,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:43,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:48,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:50,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:42:52,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:42:52,455 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:42:54,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:42:56,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:42:56,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:42:59,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:02,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:05,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:05,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:43:10,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:11,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:43:11,288 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:43:11,289 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:43:11,291 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:43:11,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:43:11,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:43:12,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:43:13,177 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:14,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:15,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:20,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:20,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:43:22,456 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:43:24,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:26,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:43:26,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:43:32,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:33,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:35,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:43:37,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:42,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:42,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:43:42,507 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:43:45,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:47,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:50,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:43:52,458 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:43:53,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:43:55,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:43:57,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:43:57,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:43:58,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:03,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:05,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:44:07,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:44:09,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:12,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:44:12,508 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:44:14,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:17,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:44:20,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:20,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:44:22,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:44:26,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:27,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:44:27,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:44:27,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:44:31,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:35,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:44:36,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:39,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:44:42,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:42,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:44:42,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:44:47,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:50,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:44:50,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:44:52,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:44:53,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:44:57,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:44:57,635 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:44:58,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:02,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:04,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:05,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:45:09,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:12,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:12,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:45:12,636 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:45:14,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:19,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:20,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:45:22,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:22,464 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:45:25,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:27,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:45:27,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:45:30,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:32,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:35,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:35,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:45:40,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:42,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:45:42,636 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:45:44,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:45,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:50,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:51,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:45:52,465 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:45:53,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:45:56,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:45:57,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:45:57,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:46:01,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:03,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:06,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:46:07,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:12,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:12,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:46:12,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:46:15,731 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:17,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:21,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:46:22,466 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:46:23,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:27,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:46:27,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:46:29,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:29,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:34,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:37,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:46:39,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:39,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:42,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:46:42,654 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:46:44,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:49,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:46:49,482 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:46:49,483 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:46:49,487 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:46:50,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:46:50,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:51,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:52,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:46:52,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:46:52,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:46:56,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:46:58,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:46:58,736 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:47:02,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:02,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:07,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:08,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:47:12,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:12,235 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:13,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:47:13,735 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:47:17,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:22,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:22,467 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:47:23,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:23,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:47:28,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:29,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:47:29,242 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:47:32,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:33,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:38,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:39,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:47:44,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:47:44,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:44,241 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:47:44,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:49,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:52,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:47:54,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:47:54,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:47:54,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:47:59,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:00,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:48:00,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:48:04,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:05,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:10,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:11,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:48:14,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:15,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:15,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:48:15,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:48:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:22,468 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:48:23,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:26,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:48:26,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:31,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:48:31,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:48:32,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:33,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:38,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:41,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:48:43,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:43,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:46,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:48:46,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:48:48,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:52,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:48:52,469 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:48:53,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:48:56,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:48:58,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:01,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:49:01,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:49:02,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:04,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:09,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:11,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:49:12,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:14,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:16,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:49:16,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:49:20,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:22,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:22,469 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:49:25,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:26,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:49:31,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:31,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:49:31,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:49:32,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:37,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:41,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:49:42,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:42,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:46,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:49:46,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:49:48,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:52,470 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:49:52,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:49:53,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:49:56,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:49:58,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:01,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:50:01,891 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:50:02,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:04,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:09,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:11,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:50:12,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:50:12,587 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:50:12,588 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:50:12,590 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:50:12,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:50:13,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:14,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:14,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:16,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:50:16,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:50:20,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:22,472 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:50:24,801 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:25,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:26,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:50:30,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:33,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:50:33,225 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:50:36,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:36,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:41,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:50:41,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:46,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:47,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:48,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:50:48,238 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:50:52,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:50:52,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:50:56,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:50:56,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:50:57,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:02,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:04,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:51:04,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:51:07,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:08,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:51:11,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:51:12,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:18,405 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:51:18,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:19,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:51:19,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:51:22,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:51:24,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:26,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-18 23:51:28,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:51:29,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:34,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:36,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:51:36,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:51:38,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:51:40,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:45,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:48,658 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:51:50,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:51,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:51:51,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:51:52,474 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:51:55,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:51:58,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:01,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:06,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:52:06,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:52:06,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:11,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:12,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:17,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:21,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:21,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:52:21,384 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:52:22,475 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:52:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:27,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:30,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:32,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:36,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:52:36,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:52:37,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:40,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:42,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:47,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:51,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:52:51,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:52:52,476 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:52:52,558 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:52:53,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:52:58,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:02,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:53:04,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:06,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:53:06,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:53:09,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:14,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:19,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:21,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:53:21,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:53:22,477 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:53:25,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:30,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:35,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:36,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:53:36,823 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:53:41,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:46,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:51,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:53:52,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:53:52,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:53:52,478 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:53:56,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:53:56,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:01,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:06,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:07,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:07,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:54:07,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:54:12,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:16,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:17,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:22,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:54:22,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:54:22,481 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:54:23,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:25,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:54:25,864 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:54:25,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:54:25,869 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:54:26,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:54:26,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:27,831 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:28,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:33,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:36,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:37,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:54:37,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:54:39,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:44,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:46,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:54:49,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:52,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:54:52,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:54:52,527 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:54:54,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:54:57,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:00,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:05,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:06,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:07,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:55:07,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:55:10,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:16,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:16,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:21,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:22,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:55:22,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:55:22,480 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:55:26,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:26,686 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:31,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:36,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:36,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:37,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:55:37,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:55:42,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:46,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:47,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:52,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:55:52,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:55:52,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:55:53,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:55:56,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:55:58,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:03,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:06,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:07,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:56:07,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:56:09,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:14,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:19,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:20,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:22,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:56:22,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:56:22,488 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:56:25,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:29,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:31,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:36,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:37,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:56:37,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:56:39,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:41,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:47,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:49,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:52,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:56:52,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:56:52,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:56:52,484 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:56:57,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:56:57,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:02,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:07,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:57:07,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:57:07,449 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:08,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:13,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:17,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:18,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:22,485 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:57:23,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:57:23,406 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:57:23,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:26,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:28,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:33,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:36,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:38,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:57:38,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:57:39,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:44,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-18 23:57:44,406 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-18 23:57:44,407 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-18 23:57:44,408 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 23:57:44,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-18 23:57:44,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:45,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:46,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:57:50,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:52,486 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:57:53,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:57:53,404 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:57:55,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:57:56,831 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:00,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:06,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:06,926 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:08,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:58:08,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:58:11,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:16,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:16,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:21,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:22,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:58:23,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:58:23,404 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:58:25,055 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:27,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:32,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:35,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:37,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:38,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:58:38,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:58:42,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:45,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:48,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:52,487 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:58:53,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:58:53,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:58:53,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:58:55,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:58:58,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:04,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:05,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:08,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:59:08,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:59:09,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:14,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:15,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:19,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:22,488 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:59:23,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:59:23,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:59:23,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:24,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:29,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:33,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:35,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:38,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:59:38,411 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:59:40,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:43,661 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:46,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:52,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 23:59:52,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-18 23:59:53,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 23:59:53,410 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-18 23:59:53,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-18 23:59:57,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:02,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:02,785 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:07,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:08,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:00:08,412 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:00:12,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:13,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:18,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:22,489 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:00:22,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:23,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:00:23,410 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:00:24,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:29,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:33,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:35,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:38,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:00:38,413 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:00:40,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:43,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:46,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:51,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:00:52,490 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:00:53,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:00:53,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:00:55,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:00:56,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:01,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:03,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:01:03,967 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:01:03,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:01:03,970 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:01:04,292 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:01:05,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:06,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:06,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:08,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:01:08,412 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:01:12,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:13,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:01:15,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:17,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:22,492 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:01:23,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:23,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:01:23,903 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:01:27,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:29,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:34,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:37,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:38,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:01:38,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:01:40,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:45,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:01:45,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:47,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:01:50,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:52,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:01:54,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:01:54,962 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:01:56,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:01:57,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:00,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:02:01,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:06,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:06,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:09,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:02:09,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:02:12,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:15,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:02:17,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:21,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:22,493 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:02:22,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:24,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:02:24,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:02:28,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:30,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:02:31,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:33,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:38,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:39,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:02:39,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:02:41,292 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:44,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:45,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:02:46,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:47,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:50,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:02:52,341 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:52,494 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:02:53,345 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:02:54,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:02:54,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:02:55,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:00,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:03:00,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:01,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:05,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:09,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:03:09,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:03:11,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:11,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:15,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:03:16,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:19,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:22,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:22,494 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:03:26,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:03:26,237 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:03:27,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:29,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:32,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:37,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:37,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:41,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:03:41,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:03:42,474 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:47,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:48,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:52,496 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:03:53,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:03:55,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:03:56,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:03:56,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:03:59,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:04,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:09,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:11,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:04:11,234 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:04:14,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:19,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:22,496 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:04:24,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:26,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:04:26,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:04:29,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:30,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:04:34,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:39,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:04:39,302 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:04:39,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:04:39,305 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:04:40,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:04:40,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:41,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:04:41,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:04:41,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:04:42,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:04:45,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:51,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:04:51,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:04:52,497 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:04:56,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:04:56,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:04:56,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:01,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:02,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:07,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:11,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:11,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:05:11,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:05:12,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:17,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:21,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:22,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:05:23,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:26,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:05:26,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:05:28,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:31,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:34,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:39,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:41,641 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:41,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:05:41,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:05:44,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:49,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:51,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:05:52,498 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:05:55,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:05:56,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:05:56,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:06:00,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:01,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:06,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:11,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:11,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:06:11,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:06:13,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:16,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:21,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:22,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:06:23,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:26,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:06:26,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:06:26,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:32,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:34,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:37,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:41,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:06:41,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:06:42,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:42,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:47,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:52,200 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:06:52,499 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:06:53,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:06:56,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:06:56,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:06:58,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:02,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:04,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:09,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:11,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:07:11,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:07:12,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:14,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:20,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:20,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:22,500 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:07:25,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:27,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:07:27,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:07:29,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:31,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:36,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:39,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:41,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:42,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:07:42,479 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:07:46,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:49,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:52,500 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:07:52,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:56,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:07:56,651 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:07:56,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:07:56,654 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:07:57,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:07:57,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:07:57,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:07:57,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:07:57,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:07:58,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:02,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:07,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:08,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:12,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:08:12,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:08:13,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:17,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:19,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:22,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:08:25,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:27,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:08:27,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:08:27,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:30,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:35,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:37,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:40,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:42,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:08:42,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:08:46,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:48,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:08:51,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:52,502 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:08:56,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:08:57,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:08:57,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:08:58,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:02,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:07,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:08,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:12,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:09:12,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:09:13,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:18,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:19,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:22,504 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:09:24,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:27,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:09:27,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:09:28,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:29,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:34,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:36,389 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:40,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:42,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:09:42,483 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:09:46,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:46,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:51,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:52,505 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:09:56,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:09:57,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:09:57,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:09:57,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:10:02,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:03,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:10:06,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:10:08,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:12,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:10:12,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:10:13,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:18,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:10:18,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:22,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:10:24,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:25,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:10:27,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:10:27,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:10:29,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:33,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:10:34,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:35,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:10:40,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:43,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:10:43,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:10:45,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:45,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:10:48,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:10:50,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:52,506 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:10:55,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:10:56,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:10:58,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:10:58,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:11:01,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:03,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:11:06,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:07,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:12,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:13,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:11:13,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:11:16,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:17,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:18,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:11:22,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:11:23,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:23,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:11:23,451 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:11:23,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:11:23,454 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:11:24,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:11:24,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:25,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:28,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:28,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:11:28,627 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:11:33,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:33,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:11:34,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:38,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:43,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:11:43,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:11:43,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:44,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:48,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:48,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:11:52,508 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:11:54,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:11:54,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:11:58,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:11:58,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:11:59,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:03,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:12:04,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:12:05,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:10,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:12,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:12:13,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:12:13,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:12:15,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:18,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:12:20,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:22,509 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:12:26,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:28,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:12:28,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:12:31,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:33,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:12:36,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:41,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:43,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:12:43,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:12:46,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:12:47,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:48,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:12:52,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:52,510 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:12:56,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:12:58,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:12:58,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:12:58,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:13:03,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:03,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:13:06,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:09,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:13,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:13:13,630 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:13:15,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:16,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:18,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:13:20,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:22,511 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:13:26,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:26,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:28,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:13:28,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:13:31,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:33,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:13:36,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:37,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:42,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:43,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:13:43,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:13:46,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:48,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:48,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:13:52,512 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:13:54,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:13:56,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:13:58,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:13:58,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:13:59,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:03,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:14:05,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:06,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:10,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:13,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:14:13,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:14:15,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:16,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:18,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:14:20,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:22,514 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:14:26,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:26,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:28,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:14:28,630 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:14:31,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:33,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:14:36,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:37,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:42,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:43,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:14:43,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:14:46,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:47,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:48,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:14:52,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:52,515 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:14:56,868 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:14:57,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:14:58,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:14:58,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:15:02,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:03,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:15:04,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:15:04,972 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:15:04,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:15:04,976 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:15:05,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:15:06,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:07,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:07,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:12,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:13,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:15:13,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:15:17,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:18,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:18,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:15:22,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:15:23,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:27,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:28,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:15:28,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:28,824 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:15:34,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:34,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:15:36,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:39,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:44,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:45,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:15:45,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:15:48,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:15:50,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:52,516 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:15:55,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:15:58,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:00,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:16:00,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:16:01,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:06,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:08,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:11,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:15,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:16:15,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:16:17,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:18,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:22,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:22,517 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:16:28,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:28,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:30,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:16:30,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:16:34,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:35,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:16:38,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:39,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:44,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:45,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:16:45,099 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:16:48,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:16:50,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:50,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:16:52,518 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:16:55,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:16:56,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:00,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:17:00,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:17:01,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:05,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:17:06,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:07,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:11,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:15,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:17:15,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:17:17,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:17,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:20,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:17:22,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:22,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:17:27,200 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:28,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:30,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:17:30,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:17:33,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:35,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:17:35,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:38,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:43,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:46,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:17:46,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:17:49,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:51,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:52,520 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:17:54,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:17:58,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:17:59,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:01,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:18:01,531 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:18:04,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:08,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:10,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:16,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:16,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:18:16,531 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:18:16,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:21,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:22,521 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:18:25,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:18:25,252 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:18:25,253 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:18:25,255 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:18:25,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:18:26,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:27,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:27,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:31,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:18:31,583 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:18:32,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:36,702 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:38,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:43,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:44,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:46,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:18:46,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:18:48,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:51,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:18:52,522 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:18:54,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:18:54,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:18:59,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:01,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:19:01,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:19:02,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:04,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:06,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:19:09,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:12,949 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:15,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:16,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:19:16,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:19:20,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:21,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:19:22,522 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:19:23,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:26,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:31,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:31,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:19:31,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:19:33,082 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:36,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:19:36,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:41,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:41,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:46,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:19:46,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:19:47,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:51,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:19:52,522 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:19:53,257 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:19:53,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:19:58,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:01,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:20:01,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:20:03,398 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:20:04,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:06,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:20:09,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:14,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:20:14,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:16,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:20:16,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:20:19,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:21,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:20:22,523 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:20:25,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:28,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:20:31,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:31,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:20:31,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:20:36,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:36,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:20:38,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:20:41,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:46,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:46,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:20:46,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:20:48,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:20:51,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:20:51,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:52,523 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:20:57,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:20:58,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:01,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:21:01,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:21:02,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:06,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:21:08,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:09,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:13,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:16,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:21:16,543 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:21:18,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:19,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:21,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:21:22,524 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:21:24,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:29,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:29,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:31,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:21:31,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:21:35,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:36,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:21:39,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:41,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:46,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:46,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:21:46,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:21:47,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:21:47,007 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:21:47,007 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:21:47,009 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:21:47,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:21:48,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:49,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:21:51,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:21:52,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:52,525 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:21:57,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:21:59,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:01,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:22:01,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:22:02,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:06,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:22:08,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:09,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:13,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:16,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:22:16,546 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:22:17,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:18,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:21,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:22:22,527 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:22:24,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:27,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:29,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:31,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:22:31,546 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:22:34,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:36,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:22:36,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:39,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:44,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:46,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:22:46,546 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:22:46,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:22:50,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:51,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:22:52,527 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:22:56,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:22:59,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:01,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:01,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:23:01,548 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:23:06,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:07,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:23:09,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:12,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:16,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:23:16,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:23:17,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:21,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:22,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:23:22,528 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:23:23,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:28,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:31,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:23:31,548 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:23:33,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:33,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:37,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:23:38,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:43,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:44,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:46,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:23:46,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:23:49,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:52,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:23:52,529 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:23:54,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:23:55,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:23:59,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:02,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:24:02,706 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:24:05,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:05,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:24:10,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:15,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:17,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:24:17,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:24:17,704 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:24:21,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:22,529 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:24:26,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:27,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:24:31,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:32,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:24:32,704 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:24:36,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:39,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:24:41,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:46,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:47,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:24:47,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:24:49,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:24:52,530 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:24:52,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:24:57,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:01,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:02,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:02,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:25:02,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:25:07,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:08,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:25:11,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:13,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:17,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:25:17,708 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:25:18,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:20,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:25:20,717 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:25:20,718 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:25:20,719 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:25:21,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:25:21,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:22,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:22,530 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:25:23,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:25:24,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:29,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:32,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:25:32,706 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:25:33,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:34,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:38,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:25:39,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:43,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:45,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:47,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:25:47,706 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:25:50,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:25:52,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:25:53,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:25:55,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:25:56,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:01,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:03,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:26:03,665 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:26:07,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:07,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:26:12,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:17,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:17,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:26:18,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:26:18,649 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:26:22,532 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:26:22,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:28,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:29,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:26:33,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:33,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:26:33,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:26:38,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:39,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:26:43,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:48,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:26:48,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:26:49,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:26:52,533 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:26:55,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:00,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:02,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:03,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:27:03,646 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:27:05,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:09,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:27:11,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:12,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:16,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:20,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:27:20,175 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:27:22,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:22,533 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:27:23,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:27,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:32,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:33,297 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:35,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:27:35,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:27:38,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:43,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:43,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:48,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:51,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:27:51,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:27:52,534 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:27:54,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:27:55,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:27:59,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:04,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:05,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:28:06,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:28:06,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:28:09,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:14,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:17,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:28:20,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:21,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:28:21,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:28:22,534 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:28:25,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:27,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:28:31,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:36,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:36,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:28:36,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:28:37,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:28:41,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:46,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:49,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:28:51,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:28:51,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:28:51,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:52,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:28:57,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:28:59,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:03,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:07,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:29:07,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:29:08,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:09,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:29:09,200 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:29:09,201 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:29:09,203 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:29:10,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:29:10,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:12,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:14,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:19,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:22,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:22,535 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:29:22,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:29:22,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:29:24,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:29,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:34,184 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:35,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:37,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:29:37,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:29:40,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:44,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:46,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:51,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:29:52,536 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:29:52,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:29:52,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:29:54,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:29:56,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:01,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:03,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:30:07,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:30:07,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:30:07,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:12,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:13,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:30:18,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:22,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:30:22,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:30:22,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:30:23,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:25,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:30:28,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:33,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:35,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:30:37,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:30:37,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:30:38,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:43,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:47,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:30:49,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:52,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:30:52,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:30:52,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:30:54,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:30:58,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:00,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:05,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:07,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:31:07,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:31:08,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:11,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:17,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:18,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:22,538 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:31:22,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:31:22,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:31:22,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:27,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:28,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:32,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:37,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:37,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:31:37,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:31:38,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:43,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:47,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:31:49,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:52,539 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:31:52,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:31:52,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:31:54,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:31:59,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:00,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:05,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:07,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:32:07,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:32:09,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:11,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:16,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:19,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:21,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:22,540 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:32:22,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:32:22,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:32:26,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:27,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:32:29,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:32,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:37,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:38,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:32:38,332 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:32:39,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:32:39,709 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:32:39,710 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:32:39,713 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:32:39,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:32:40,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:41,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:42,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:47,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:52,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:32:52,540 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:32:53,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:32:53,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:32:53,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:32:58,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:02,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:33:03,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:08,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:33:08,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:33:09,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:14,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:18,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:33:19,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:22,541 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:33:25,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:33:25,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:25,039 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:33:28,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:33:30,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:35,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:38,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:33:40,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:33:40,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:33:40,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:45,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:50,471 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:33:51,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:33:52,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:33:55,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:33:55,080 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:33:56,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:00,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:01,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:06,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:10,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:34:10,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:34:10,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:12,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:17,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:19,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:22,542 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:34:22,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:25,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:34:25,080 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:34:28,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:31,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:33,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:38,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:40,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:34:40,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:34:41,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:44,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:49,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:51,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:34:52,543 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:34:54,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:34:55,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:34:55,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:35:00,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:02,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:35:05,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:10,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:35:10,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:35:11,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:16,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:21,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:22,543 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:35:25,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:35:25,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:35:26,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:30,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:35:32,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:37,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:40,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:35:40,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:35:42,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:42,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:35:47,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:52,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:35:52,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:35:52,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:35:55,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:35:55,081 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:35:58,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:02,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:03,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:08,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:10,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:36:10,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:36:12,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:14,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:19,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:22,544 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:36:24,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:24,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:25,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:36:25,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:36:30,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:34,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:36:34,121 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:36:34,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:36:34,124 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:36:34,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:36:34,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:36,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:36,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:40,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:36:40,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:36:41,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:45,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:36:46,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:51,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:52,545 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:36:55,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:36:55,083 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:36:57,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:36:58,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:02,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:07,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:08,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:10,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:37:10,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:37:13,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:15,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:37:18,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:19,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:22,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:37:24,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:26,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:37:26,308 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:37:30,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:30,261 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:35,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:40,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:40,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:41,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:37:41,305 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:37:46,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:51,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:37:52,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:37:52,546 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:37:56,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:37:56,304 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:37:56,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:01,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:04,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:38:07,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:11,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:38:11,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:38:13,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:14,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:38:18,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:22,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:38:23,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:24,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:38:26,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:38:26,867 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:38:29,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:34,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:36,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:38:39,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:41,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:38:41,870 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:38:44,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:46,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:38:49,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:52,548 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:38:55,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:38:56,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:38:56,867 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:38:56,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:01,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:06,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:09,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:11,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:11,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:39:11,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:39:16,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:19,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:21,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:22,549 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:39:26,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:39:26,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:39:27,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:28,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:32,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:37,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:40,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:41,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:39:41,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:39:43,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:48,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:50,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:39:52,549 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:39:53,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:39:56,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:39:56,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:39:59,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:00,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:05,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:10,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:10,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:40:10,292 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:40:10,293 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:40:10,297 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:40:10,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:40:11,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:11,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:40:11,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:40:12,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:15,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:20,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:22,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:22,550 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:40:25,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:26,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:40:26,869 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:40:31,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:32,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:36,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:41,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:41,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:40:41,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:40:44,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:46,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:51,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:40:52,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:40:54,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:40:56,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:40:56,869 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:40:57,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:02,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:02,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:41:04,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:07,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:12,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:41:12,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:41:12,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:14,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:18,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:22,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:41:24,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:27,001 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:27,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:41:27,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:41:29,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:33,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:41:34,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:37,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:40,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:42,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:41:42,613 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:41:45,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:47,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:48,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:41:50,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:52,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:41:56,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:41:57,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:41:57,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:41:57,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:42:01,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:03,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:42:07,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:09,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:42:12,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:42:12,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:42:12,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:18,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:18,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:42:18,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:42:22,553 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:42:23,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:27,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:42:27,613 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:42:28,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:42:28,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:33,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:33,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:42:38,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:42:38,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:42,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:42:42,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:42:43,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:48,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:42:49,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:50,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:42:52,554 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:42:55,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:42:57,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:42:57,734 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:43:00,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:00,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:03,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:43:05,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:10,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:11,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:12,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:43:12,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:43:16,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:18,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:43:20,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:22,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:22,554 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:43:27,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:28,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:43:28,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:43:30,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:33,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:33,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:43:38,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:39,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:43:39,852 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:43:39,853 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:43:39,855 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:43:39,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:43:40,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:41,947 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:43,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:43:43,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:43:44,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:49,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:43:49,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:52,555 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:43:53,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:43:54,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:43:59,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:43:59,560 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:43:59,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:03,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:04,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:09,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:13,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:14,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:44:14,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:44:15,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:20,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:44:20,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:22,556 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:44:23,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:26,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:30,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:44:30,724 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:44:31,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:33,311 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:36,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:41,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:45,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:45,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:44:45,726 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:44:47,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:52,561 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:44:53,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:44:55,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:44:58,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:02,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:45:02,305 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:45:03,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:05,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:45:08,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:13,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:14,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:45:17,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:45:17,305 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:45:19,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:22,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:45:24,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:45:25,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:30,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:33,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:45:33,644 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:45:35,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:36,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:45:40,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:45,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:46,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:45:48,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:45:48,709 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:45:50,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:52,562 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:45:56,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:45:56,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:01,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:03,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:46:03,711 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:46:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:09,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:12,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:17,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:18,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:46:18,710 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:46:19,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:22,563 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:46:22,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:27,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:29,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:33,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:33,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:46:33,709 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:46:38,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:41,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:43,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:48,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:46:48,711 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:46:49,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:51,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:46:52,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:46:54,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:46:59,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:01,404 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:03,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:47:03,709 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:47:04,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:09,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:10,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:47:10,146 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:47:10,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:47:10,148 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:47:10,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:47:11,497 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:15,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:19,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:47:19,542 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:47:20,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:22,564 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:47:23,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:26,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:32,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:33,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:34,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:47:34,538 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:47:37,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:42,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:43,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:48,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:50,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:47:50,806 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:47:52,565 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:47:52,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:47:54,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:47:59,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:02,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:05,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:05,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:48:05,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:48:11,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:13,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:16,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:21,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:21,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:48:21,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:48:21,836 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:48:22,565 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:48:25,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:27,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:32,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:35,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:36,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:48:36,834 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:48:37,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:42,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:45,257 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:47,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:52,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:48:52,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:48:52,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:48:52,598 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:48:52,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:48:55,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:48:58,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:03,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:07,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:49:07,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:49:07,595 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:49:09,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:11,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:49:14,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:19,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:21,509 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:49:22,567 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:49:22,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:49:22,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:49:24,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:30,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:31,596 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:49:35,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:38,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:49:38,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:49:38,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:49:40,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:41,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:49:45,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:51,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:49:51,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:49:52,568 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:49:53,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:49:53,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:49:53,574 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:49:56,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:02,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:03,807 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:07,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:08,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:50:08,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:50:08,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:50:12,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:13,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:17,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:22,579 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:50:23,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:50:23,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:50:23,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:50:23,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:23,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:28,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:34,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:34,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:38,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:50:38,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:50:38,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:50:39,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:42,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:50:42,098 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:50:42,098 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:50:42,101 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:50:43,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:50:43,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:44,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:45,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:50,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:50:52,569 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:50:53,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:50:53,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:50:53,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:50:55,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:50:55,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:00,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:05,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:05,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:08,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:51:08,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:51:08,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:51:10,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:15,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:17,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:20,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:22,570 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:51:24,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:51:24,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:51:24,033 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:51:26,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:27,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:31,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:36,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:37,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:39,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:51:39,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:51:39,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:51:42,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:47,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:49,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:52,571 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:51:52,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:51:53,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:51:54,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:51:54,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:51:54,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:51:58,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:01,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:03,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:08,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:09,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:52:09,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:52:09,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:52:11,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:14,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:19,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:22,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:22,572 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:52:24,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:52:24,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:52:25,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:30,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:34,085 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:35,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:40,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:52:40,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:52:40,248 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:52:41,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:44,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:46,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:51,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:52:52,572 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:52:53,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:52:55,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:52:55,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:52:55,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:52:57,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:02,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:05,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:08,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:10,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:53:10,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:53:10,249 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:53:13,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:15,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:18,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:22,574 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:53:23,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:25,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:53:25,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:53:25,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:29,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:35,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:35,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:40,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:40,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:53:40,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:53:40,249 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:53:45,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:45,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:50,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:53:52,575 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:53:55,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:53:55,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:53:55,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:53:56,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:01,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:05,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:07,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:11,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:54:11,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:54:11,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:54:12,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:12,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:54:12,446 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:54:12,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:54:12,448 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:54:12,821 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:54:13,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:17,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:22,575 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:54:22,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:23,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:26,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:54:26,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:54:26,242 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:54:28,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:33,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:34,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:39,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:41,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:54:41,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:54:41,241 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:54:42,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:44,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:49,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:52,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:54:52,576 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:54:54,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:54:56,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:54:56,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:54:56,246 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:55:00,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:02,182 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:05,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:10,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:11,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:55:11,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:55:11,242 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:55:12,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:16,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:22,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:22,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:22,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:55:26,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:55:26,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:55:26,246 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:55:27,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:32,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:33,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:38,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:39,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:41,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:55:41,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:55:41,242 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:55:43,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:48,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:49,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:55:52,577 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:55:53,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:55:57,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:55:57,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:55:57,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:55:59,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:01,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:04,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:09,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:11,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:12,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:56:12,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:56:12,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:56:15,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:20,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:21,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:22,578 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:56:25,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:28,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:56:28,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:56:28,022 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:56:31,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:31,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:36,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:41,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:42,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:43,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:56:43,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:56:43,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:56:48,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:49,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:56:52,578 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:56:53,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:56:58,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:56:58,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:56:58,022 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:56:59,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:00,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:04,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:09,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:10,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:13,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:57:13,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:57:15,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:18,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:20,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:22,578 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:57:26,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:26,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 00:57:26,467 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 00:57:26,467 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 00:57:26,469 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 00:57:27,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 00:57:28,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:57:28,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:57:28,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:32,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:37,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:38,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:42,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:44,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:57:44,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:57:44,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:57:48,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:48,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:52,581 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:57:54,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:58,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:57:59,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:57:59,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:57:59,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:57:59,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:58:04,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:06,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:10,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:14,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:58:14,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:58:14,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:58:15,637 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:15,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:20,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:22,582 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:58:25,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:26,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:29,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:58:29,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:58:29,609 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:58:31,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:35,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:37,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:42,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:44,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:58:44,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:58:44,609 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:58:45,842 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:47,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:52,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:58:53,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:53,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:58:58,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:58:59,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:58:59,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:58:59,609 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:59:03,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:03,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:08,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:14,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:14,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:14,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:59:14,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:59:14,610 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:59:19,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:22,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:59:24,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:25,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:30,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 00:59:30,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:59:30,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:59:30,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:34,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:36,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:42,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:42,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:45,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 00:59:45,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 00:59:47,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:52,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 00:59:52,583 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 00:59:52,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 00:59:57,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:00,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:00:00,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:00:02,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:03,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:08,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:12,471 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:13,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:15,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:00:15,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:00:19,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:20,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:22,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:00:24,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:29,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:30,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:00:30,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:00:30,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:00:30,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:35,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:38,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:00:38,198 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:00:38,199 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:00:38,202 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:00:38,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:00:40,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:41,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:45,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:00:45,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:00:45,353 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:00:46,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:50,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:00:52,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:00:52,584 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:00:57,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:00,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:01:00,351 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:01:00,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:03,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:08,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:09,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:13,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:15,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:01:15,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:01:15,760 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:01:17,946 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:19,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:22,585 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:01:24,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:28,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:30,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:30,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:01:30,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:01:30,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:01:36,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:38,103 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:41,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:45,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:01:45,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:01:45,762 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:01:46,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:48,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:01:51,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:52,585 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:01:57,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:01:58,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:00,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:02:00,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:02:00,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:02:02,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:06,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:08,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:13,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:15,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:02:15,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:02:15,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:02:16,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:18,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:19,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:22,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:02:24,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:28,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:30,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:30,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:02:30,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:02:30,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:02:35,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:38,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:41,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:45,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:02:45,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:02:45,763 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:02:47,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:48,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:02:52,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:52,587 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:02:58,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:02:58,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:00,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:03:00,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:03:00,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:03:03,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:06,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:09,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:14,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:15,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:03:15,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:03:15,761 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:03:18,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:20,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:22,598 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:03:25,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:28,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:30,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:03:30,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:03:30,760 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:03:30,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:35,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:40,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:41,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:46,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:47,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:03:47,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:03:47,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:03:50,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:03:51,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:03:52,588 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:03:56,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:01,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:02,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:04:02,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:04:02,169 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:04:02,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:04:02,376 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:04:02,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:04:02,379 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:04:02,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:04:04,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:04:07,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:12,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:14,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:04:17,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:04:17,162 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:04:18,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:22,589 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:04:23,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:26,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:04:29,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:32,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:04:32,162 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:04:34,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:38,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:04:39,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:44,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:47,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:04:47,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:04:47,241 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:04:50,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:04:51,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:04:52,590 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:04:56,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:02,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:05:02,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:05:02,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:02,165 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:05:02,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:05:07,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:12,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:05:12,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:17,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:05:17,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:05:17,165 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:05:18,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:22,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:05:23,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:24,474 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:05:28,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:32,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:05:32,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:05:32,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:05:34,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:36,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:05:39,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:44,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:47,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:05:47,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:05:47,536 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:05:48,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:05:49,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:52,591 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:05:55,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:05:58,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:01,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:02,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:06:02,480 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:06:06,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:10,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:12,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:17,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:17,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:06:17,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:06:17,483 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:06:20,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:22,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:22,592 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:06:27,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:31,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:32,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:06:32,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:06:32,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:06:32,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:37,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:43,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:43,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:47,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:06:47,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:06:48,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:52,592 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:06:54,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:06:55,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:06:59,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:02,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:07:02,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:07:02,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:07:04,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:05,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:07:09,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:15,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:07:15,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:17,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:07:17,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:07:17,491 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:07:20,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:22,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:07:25,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:26,442 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:07:30,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:32,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:07:32,491 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:07:36,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:38,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:07:41,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:46,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:46,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:07:46,855 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:07:46,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:07:46,859 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:07:47,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:07:47,491 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:07:47,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:07:48,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:07:51,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:52,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:07:57,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:07:58,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:02,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:02,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:08:02,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:08:02,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:08:07,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:08,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:13,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:17,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:08:17,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:08:17,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:08:18,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:20,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:22,594 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:08:23,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:28,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:31,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:32,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:08:32,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:08:34,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:39,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:43,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:44,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:47,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:08:47,873 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:08:50,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:08:52,595 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:08:53,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:08:55,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:00,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:02,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:09:02,871 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:09:06,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:11,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:16,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:17,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:09:17,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:09:17,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:09:22,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:22,597 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:09:27,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:09:27,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:32,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:32,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:09:32,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:09:38,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:39,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:09:43,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:47,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:09:47,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:09:49,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:49,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:09:52,598 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:09:54,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:09:59,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:00,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:02,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:10:02,902 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:10:05,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:10,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:10,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:15,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:19,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:10:19,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:10:19,025 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:10:20,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:21,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:22,598 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:10:26,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:32,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:32,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:34,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:10:34,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:10:34,024 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:10:37,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:42,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:42,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:47,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:49,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:10:49,024 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:10:52,600 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:10:52,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:10:55,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:10:58,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:03,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:04,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:11:04,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:11:04,667 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:11:07,318 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:11:09,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:14,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:19,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:11:19,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:19,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:11:19,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:11:19,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:11:22,600 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:11:25,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:29,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:11:30,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:34,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:11:34,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:11:34,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:11:35,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:38,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:11:38,369 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:11:38,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:11:38,374 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:11:38,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:11:39,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:11:41,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:46,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:49,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:11:49,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:11:49,661 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:11:51,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:11:51,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:11:52,600 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:11:57,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:00,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:02,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:04,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:12:04,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:12:04,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:12:08,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:12,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:14,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:19,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:20,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:12:20,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:12:20,032 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:12:22,601 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:12:22,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:25,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:30,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:35,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:12:35,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:12:35,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:36,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:41,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:45,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:46,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:50,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:12:50,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:12:52,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:12:52,602 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:12:55,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:12:58,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:03,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:05,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:13:05,020 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:13:07,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:13:08,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:13,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:17,359 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:13:18,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:20,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:13:20,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:13:22,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:13:23,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:27,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:13:29,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:34,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:35,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:13:35,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:13:35,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:13:39,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:13:39,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:44,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:49,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:13:50,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:50,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:13:50,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:13:52,603 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:13:55,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:13:59,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:00,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:05,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:06,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:14:06,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:14:06,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:14:09,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:11,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:16,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:21,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:21,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:21,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:14:21,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:14:21,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:14:22,604 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:14:27,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:31,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:33,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:36,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:14:36,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:14:36,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:14:39,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:40,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:44,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:49,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:51,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:14:51,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:14:51,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:14:51,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:14:52,604 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:14:54,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:14:59,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:03,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:15:04,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:06,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:15:06,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:15:06,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:15:10,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:12,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:15:12,497 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:15:12,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:15:12,501 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:15:13,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:15:13,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:15:15,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:15:15,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:20,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:21,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:15:21,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:15:21,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:15:22,605 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:15:25,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:30,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:35,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:36,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:15:36,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:15:36,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:15:41,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:46,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:51,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:15:51,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:15:51,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:15:51,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:15:52,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:15:56,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:01,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:06,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:06,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:16:06,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:16:06,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:16:12,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:17,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:21,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:16:21,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:16:21,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:16:22,606 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:16:22,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:27,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:32,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:36,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:16:36,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:16:36,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:16:38,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:43,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:48,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:51,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:16:51,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:16:51,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:16:52,607 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:16:53,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:16:58,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:03,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:06,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:17:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:17:06,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:17:09,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:14,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:19,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:21,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:17:21,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:17:21,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:17:22,607 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:17:24,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:29,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:34,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:36,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:17:36,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:17:36,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:17:40,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:45,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:50,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:17:51,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:17:51,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:17:51,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:17:52,607 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:17:55,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:00,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:05,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:06,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:18:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:18:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:18:11,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:16,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:21,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:21,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:18:21,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:18:21,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:18:22,608 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:18:26,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:31,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:36,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:36,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:18:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:18:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:18:42,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:47,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:51,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:18:51,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:18:51,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:18:52,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:18:52,609 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:18:57,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:02,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:06,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:19:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:19:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:19:08,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:13,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:18,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:21,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:19:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:19:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:19:22,619 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:19:23,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:28,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:33,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:36,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:19:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:19:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:19:39,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:44,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:49,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:51,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:19:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:19:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:19:52,610 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:19:54,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:19:59,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:04,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:06,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:20:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:20:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:20:10,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:15,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:20,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:21,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:20:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:20:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:20:22,610 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:20:25,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:30,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:35,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:36,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:20:36,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:20:36,869 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:20:41,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:46,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:51,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:20:51,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:20:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:20:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:20:52,610 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:20:56,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:01,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:06,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:06,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:21:06,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:21:06,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:21:12,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:17,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:21,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:21:21,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:21:21,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:21:22,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:21:22,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:27,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:32,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:36,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:21:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:21:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:21:38,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:43,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:48,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:51,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:21:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:21:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:21:52,611 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:21:53,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:21:58,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:03,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:06,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:22:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:22:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:22:09,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:14,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:19,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:21,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:22:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:22:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:22:22,612 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:22:24,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:29,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:34,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:36,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:22:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:22:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:22:40,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:45,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:50,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:22:51,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:22:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:22:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:22:52,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:22:55,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:00,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:05,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:06,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:23:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:23:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:23:11,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:16,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:21,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:21,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:23:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:23:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:23:22,613 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:23:26,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:31,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:36,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:36,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:23:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:23:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:23:42,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:47,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:51,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:23:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:23:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:23:52,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:23:52,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:23:57,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:02,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:06,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:24:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:24:06,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:24:08,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:13,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:18,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:21,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:24:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:24:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:24:22,614 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:24:23,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:28,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:33,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:36,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:24:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:24:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:24:39,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:44,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:49,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:51,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:24:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:24:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:24:52,615 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:24:54,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:24:59,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:04,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:06,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:25:06,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:25:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:25:10,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:15,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:20,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:21,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:25:21,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:25:21,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:25:22,615 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:25:25,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:30,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:35,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:36,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:25:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:25:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:25:41,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:46,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:51,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:25:51,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:25:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:25:51,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:25:52,617 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:25:56,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:01,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:06,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:06,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:26:06,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:26:06,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:26:12,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:17,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:21,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:26:21,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:26:21,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:26:22,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:22,616 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:26:27,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:32,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:36,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:26:36,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:26:36,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:26:38,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:43,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:48,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:51,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:26:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:26:51,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:26:52,617 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:26:53,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:26:58,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:03,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:06,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:27:06,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:27:06,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:27:09,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:14,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:19,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:21,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:27:21,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:27:21,936 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:27:22,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:27:22,262 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:27:22,263 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:27:22,266 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:27:22,617 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:27:22,840 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:27:23,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:27:24,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:24,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:27:29,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:34,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:36,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:27:36,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:27:36,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:27:40,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:45,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:50,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:27:51,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:27:51,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:27:51,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:27:52,618 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:27:54,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:27:55,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:00,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:05,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:06,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:28:07,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:28:07,490 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:28:10,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:15,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:20,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:22,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:28:22,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:28:22,561 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:28:22,719 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:28:26,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:31,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:36,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:38,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:28:38,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:28:38,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:28:40,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:28:41,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:44,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:28:46,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:28:47,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:52,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:52,619 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:28:57,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:28:59,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:29:02,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:07,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:07,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:29:07,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:29:07,637 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:29:08,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:29:12,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:14,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:29:14,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:29:18,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:22,620 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:29:23,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:28,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:29,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:29:29,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:29:33,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:38,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:43,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:44,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:29:44,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:29:49,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:52,620 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:29:54,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:59,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:29:59,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:29:59,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:30:05,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:10,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:14,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:30:14,831 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:30:16,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:18,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:30:21,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:22,621 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:30:27,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:29,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:30:29,831 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:30:30,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:30:33,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:38,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:43,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:30:43,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:44,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:30:44,831 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:30:49,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:52,622 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:30:54,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:55,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:30:59,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:30:59,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:30:59,831 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:31:05,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:06,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:31:10,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:14,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:31:14,833 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:31:15,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:16,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:31:20,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:22,624 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:31:25,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:28,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:31:29,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:31:29,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:31:29,894 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:31:31,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:36,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:40,502 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:31:41,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:44,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:31:44,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:31:44,893 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:31:47,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:50,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:31:52,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:31:52,624 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:31:57,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:00,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:32:00,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:32:00,720 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:32:02,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:04,836 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:32:08,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:13,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:15,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:32:15,716 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:32:16,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:32:18,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:22,624 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:32:24,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:29,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:32:30,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:30,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:32:30,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:32:30,741 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:32:36,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:41,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:32:41,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:45,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:32:45,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:32:45,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:32:46,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:51,286 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:32:52,625 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:32:52,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:32:57,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:00,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:33:00,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:33:00,742 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:33:02,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:33:03,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:08,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:13,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:14,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:33:15,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:33:15,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:33:15,741 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:33:18,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:22,627 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:33:24,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:26,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:33:29,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:30,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:33:30,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:33:30,744 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:33:35,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:36,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:33:40,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:45,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:45,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:33:45,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:33:45,741 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:33:46,982 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:33:50,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:52,628 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:33:56,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:33:57,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:00,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:34:00,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:34:00,741 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:34:01,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:05,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:34:05,465 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:34:05,466 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:34:05,469 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:34:06,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:34:07,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:07,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:12,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:15,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:34:15,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:34:15,743 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:34:17,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:17,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:22,630 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:34:23,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:27,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:28,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:30,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:34:30,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:34:30,742 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:34:34,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:39,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:40,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:45,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:45,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:34:45,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:34:45,745 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:34:48,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:34:50,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:52,630 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:34:56,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:34:58,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:35:00,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:35:00,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:35:00,745 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:35:01,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:06,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:08,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:35:12,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:16,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:35:16,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:35:16,521 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:35:17,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:18,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:35:22,630 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:35:23,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:28,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:31,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:35:31,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:35:31,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:35:33,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:39,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:35:39,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:44,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:46,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:35:46,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:35:46,522 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:35:49,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:35:49,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:35:52,631 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:35:55,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:01,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:01,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:01,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:36:01,532 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:36:06,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:11,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:12,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:16,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:36:16,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:36:16,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:36:17,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:21,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:22,632 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:36:23,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:28,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:31,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:36:31,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:36:33,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:33,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:38,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:43,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:43,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:46,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:36:46,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:36:49,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:36:52,632 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:36:52,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:36:55,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:00,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:01,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:37:01,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:37:01,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:37:02,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:06,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:11,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:14,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:16,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:16,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:37:16,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:37:21,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:22,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:37:25,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:26,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:31,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:32,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:37:32,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:37:32,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:37:35,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:36,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:41,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:45,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:37:45,503 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:37:45,504 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:37:45,507 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:37:46,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:37:47,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:47,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:47,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:37:47,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:37:52,634 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:37:52,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:37:57,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:37:58,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:02,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:38:02,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:38:04,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:09,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:38:09,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:14,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:17,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:38:17,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:38:17,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:38:19,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:38:20,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:22,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:38:26,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:31,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:31,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:38:32,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:38:32,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:38:36,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:42,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:43,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:38:47,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:38:47,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:38:47,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:52,635 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:38:53,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:38:53,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:38:58,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:02,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:39:02,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:39:03,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:04,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:39:08,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:13,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:15,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:39:17,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:39:17,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:39:19,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:22,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:39:24,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:27,788 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:39:29,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:32,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:39:32,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:39:35,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:39,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:39:41,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:46,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:47,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:39:47,740 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:39:50,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:39:52,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:39:52,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:39:57,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:01,055 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:02,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:40:02,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:40:02,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:07,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:11,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:13,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:18,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:18,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:40:18,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:40:18,833 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:40:21,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:22,638 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:40:24,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:29,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:31,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:33,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:40:33,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:40:33,832 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:40:35,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:40,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:41,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:45,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:50,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:40:50,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:40:50,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:40:51,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:40:52,639 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:40:53,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:40:56,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:02,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:03,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:05,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:41:05,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:41:05,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:41:08,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:13,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:14,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:19,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:20,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:41:20,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:41:20,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:41:20,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:41:20,866 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:41:20,867 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:41:20,868 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:41:21,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:41:21,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:22,640 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:41:22,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:24,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:29,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:31,905 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:34,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:35,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:41:35,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:41:35,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:41:40,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:41,982 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:45,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:50,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:41:50,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:41:51,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:41:52,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:41:52,640 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:41:57,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:02,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:03,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:05,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:42:05,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:42:08,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:11,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:14,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:19,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:20,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:42:20,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:42:21,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:22,640 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:42:24,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:30,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:31,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:35,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:42:35,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:42:36,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:41,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:41,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:46,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:50,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:42:50,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:42:50,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:42:51,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:42:52,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:42:52,641 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:42:57,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:01,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:02,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:05,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:43:05,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:43:08,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:11,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:14,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:19,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:20,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:43:20,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:43:21,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:22,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:43:24,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:29,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:31,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:34,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:35,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:43:35,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:43:40,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:41,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:45,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:50,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:43:50,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:43:50,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:43:51,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:43:52,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:43:52,642 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:43:56,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:02,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:02,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:05,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:44:05,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:44:08,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:12,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:13,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:18,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:20,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:44:20,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:44:20,212 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:44:22,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:22,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:44:24,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:29,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:31,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:34,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:35,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:44:35,210 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:44:38,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:44:38,788 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:44:38,788 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:44:38,791 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:44:39,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:44:39,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:39,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:41,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:44,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:49,484 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:44:50,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:44:50,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:44:50,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:52,643 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:44:55,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:44:59,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:01,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:05,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:45:05,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:45:05,214 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:45:07,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:07,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:12,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:17,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:17,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:20,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:45:20,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:45:22,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:45:23,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:27,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:29,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:34,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:35,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:45:35,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:45:35,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:39,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:45,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:45,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:45:50,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:45:50,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:45:50,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:52,646 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:45:55,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:45:55,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:00,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:05,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:46:05,211 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:46:06,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:06,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:11,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:14,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:17,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:20,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:46:20,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:46:20,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:46:22,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:22,646 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:46:22,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:27,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:32,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:33,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:35,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:46:35,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:46:38,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:42,220 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:43,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:48,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:50,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:46:50,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:46:50,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:46:52,646 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:46:53,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:46:59,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:00,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:04,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:06,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:47:06,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:47:06,305 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:47:08,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:09,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:14,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:18,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:20,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:21,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:47:21,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:47:21,304 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:47:22,648 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:47:26,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:26,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:31,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:35,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:36,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:47:36,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:47:36,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:36,304 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:47:41,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:43,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:47:43,816 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:47:43,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:47:43,819 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:47:44,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:47:45,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:46,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:51,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:47:51,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:47:51,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:47:52,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:47:52,649 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:47:53,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:47:57,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:03,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:03,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:06,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:48:06,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:48:06,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:48:09,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:11,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:14,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:19,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:21,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:48:21,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:48:21,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:48:22,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:22,649 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:48:25,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:30,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:31,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:36,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:37,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:48:37,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:48:37,306 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:48:40,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:41,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:46,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:48,242 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:48:51,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:52,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:48:52,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:48:52,307 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:48:52,650 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:48:56,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:48:58,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:01,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:06,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:06,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:07,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:49:07,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:49:07,306 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:49:12,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:16,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:17,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:22,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:49:22,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:49:22,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:49:22,804 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:49:22,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:26,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:28,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:34,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:34,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:37,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:49:37,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:49:37,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:49:39,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:42,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:45,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:51,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:49:52,651 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:49:52,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:49:52,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:49:52,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:49:52,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:49:56,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:00,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:02,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:07,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:07,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:50:07,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:50:07,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:50:13,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:15,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:18,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:22,653 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:50:22,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:50:22,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:50:22,804 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:50:23,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:25,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:28,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:33,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:34,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:37,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:50:37,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:50:37,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:50:39,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:44,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:45,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:50,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:50:51,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:50:51,110 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:50:51,111 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:50:51,114 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:50:52,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:50:52,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:50:52,653 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:50:52,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:50:52,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:50:52,803 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:50:55,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:01,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:02,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:06,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:08,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:51:08,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:51:08,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:51:10,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:11,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:16,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:20,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:21,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:22,654 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:51:23,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:51:23,533 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:51:27,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:30,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:32,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:38,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:38,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:38,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:51:38,534 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:51:43,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:48,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:51:49,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:52,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:51:54,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:51:54,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:51:54,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:51:54,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:51:56,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:00,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:05,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:06,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:09,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:52:09,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:52:09,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:52:11,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:14,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:16,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:21,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:22,661 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:52:24,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:52:24,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:52:24,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:27,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:33,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:34,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:38,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:40,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:52:40,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:52:40,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:52:42,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:43,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:48,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:52,657 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:52:52,849 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:52:53,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:52:55,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:52:55,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:52:55,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:52:59,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:00,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:04,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:09,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:10,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:53:10,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:53:10,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:53:10,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:15,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:19,004 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:20,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:22,658 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:53:25,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:25,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:53:25,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:53:25,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:53:29,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:30,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:35,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:38,136 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:40,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:53:40,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:53:40,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:53:40,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:46,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:46,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:51,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:53:52,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:53:54,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:53:54,666 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:53:54,666 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:53:54,668 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:53:55,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:53:55,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:53:55,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:53:55,670 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:53:56,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:53:56,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:01,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:04,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:07,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:11,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:54:11,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:54:11,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:54:13,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:14,371 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:18,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:22,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:54:23,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:24,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:26,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:54:26,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:54:29,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:32,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:34,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:39,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:41,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:54:41,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:54:42,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:45,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:50,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:52,659 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:54:52,666 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:54:55,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:54:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:54:56,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:54:56,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:55:00,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:00,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:05,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:10,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:11,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:11,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:55:11,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:55:17,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:18,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:22,660 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:55:22,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:55:26,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:55:28,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:28,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:33,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:36,965 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:38,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:41,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:55:41,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:55:44,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:47,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:49,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:52,661 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:55:55,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:55:55,094 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:55:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:55:56,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:56:00,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:05,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:05,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:10,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:11,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:56:11,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:56:13,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:16,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:21,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:22,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:22,661 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:56:26,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:56:26,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:56:30,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:32,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:37,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:40,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:41,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:56:41,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:56:41,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:56:43,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:48,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:50,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:52,665 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:56:53,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:56:56,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:56:56,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:56:56,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:56:57,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 01:56:57,689 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 01:56:57,690 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 01:56:57,693 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 01:56:58,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 01:56:58,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:56:58,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:03,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:08,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:09,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:11,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:57:11,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:57:11,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:57:15,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:18,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:20,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:22,663 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:57:26,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:26,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:26,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:57:26,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:57:26,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:57:31,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:36,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:36,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:41,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:41,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:57:41,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:57:41,944 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:57:46,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:47,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:52,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:57:52,663 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:57:55,001 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:57:56,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:57:56,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:57:56,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:57:58,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:04,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:05,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:09,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:11,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:58:11,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:58:11,944 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:58:13,125 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:14,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:19,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:22,663 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:58:23,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:24,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:26,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:58:26,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:58:26,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:58:30,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:31,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:35,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:40,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:41,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:41,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:58:41,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:58:41,944 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:58:45,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:50,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:58:51,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:52,664 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:58:56,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:58:56,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:58:56,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:58:56,945 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:58:58,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:02,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:07,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:08,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:11,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:59:11,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:59:11,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:59:12,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:16,920 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:17,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:22,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:22,665 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:59:26,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:59:26,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:59:26,944 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:59:26,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:28,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:33,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:37,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:39,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:43,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:59:43,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:59:43,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 01:59:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:45,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:49,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:52,665 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 01:59:54,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 01:59:55,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 01:59:58,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 01:59:58,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 01:59:58,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:00:00,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:02,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:00:02,276 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:00:02,277 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:00:02,278 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:00:02,686 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:00:02,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:04,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:06,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:12,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:12,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:13,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:00:13,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:00:17,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:20,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:22,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:00:22,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:27,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:28,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:00:28,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:00:28,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:00:30,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:33,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:38,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:38,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:43,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:43,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:00:43,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:00:48,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:48,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:00:52,666 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:00:54,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:00:58,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:00:58,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:00:59,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:00,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:05,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:07,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:11,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:13,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:01:13,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:01:13,848 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:01:16,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:17,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:21,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:22,668 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:01:25,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:27,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:28,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:01:28,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:01:33,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:35,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:38,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:43,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:43,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:43,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:01:43,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:01:49,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:52,667 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:01:53,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:01:54,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:01:59,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:01:59,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:01:59,520 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:01:59,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:01,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:05,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:10,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:11,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:14,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:02:14,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:02:14,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:02:15,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:19,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:20,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:22,668 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:02:26,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:29,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:02:29,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:02:29,519 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:02:29,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:31,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:36,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:39,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:42,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:45,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:02:45,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:02:45,379 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:02:47,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:47,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:52,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:02:53,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:02:56,849 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:02:58,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:00,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:03:00,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:03:04,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:04,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:03:04,884 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:03:04,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:03:04,887 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:03:04,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:03:06,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:14,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:14,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:15,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:03:15,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:03:20,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:22,669 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:03:23,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:25,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:30,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:32,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:03:32,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:03:32,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:03:35,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:36,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:41,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:46,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:03:47,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:03:47,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:03:49,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:52,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:03:52,670 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:03:57,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:03:57,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:02,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:04:02,018 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:04:03,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:07,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:08,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:13,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:15,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:17,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:04:17,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:04:17,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:04:19,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:22,670 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:04:24,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:25,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:29,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:32,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:04:32,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:04:32,021 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:04:33,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:34,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:39,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:43,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:45,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:47,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:04:47,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:04:50,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:04:51,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:04:52,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:04:56,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:01,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:01,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:02,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:05:02,019 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:05:07,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:09,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:12,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:17,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:05:17,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:05:17,029 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:05:18,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:19,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:22,671 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:05:23,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:29,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:29,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:32,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:05:32,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:05:35,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:37,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:40,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:45,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:47,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:05:47,028 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:05:47,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:51,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:05:52,673 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:05:56,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:05:56,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:01,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:02,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:06:02,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:06:02,647 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:06:05,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:06,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:11,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:12,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:06:12,665 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:06:12,666 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:06:12,668 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:06:13,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:06:13,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:15,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:17,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:06:17,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:06:17,646 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:06:17,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:22,673 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:06:23,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:23,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:28,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:32,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:06:32,646 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:06:33,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:33,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:38,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:41,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:44,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:48,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:06:48,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:06:48,603 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:06:49,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:06:51,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:06:52,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:06:55,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:01,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:01,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:03,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:07:03,602 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:07:06,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:09,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:12,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:17,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:18,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:07:18,602 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:07:19,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:22,674 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:07:23,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:27,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:28,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:33,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:34,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:07:34,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:07:34,585 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:07:37,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:39,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:44,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:45,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:07:49,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:07:49,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:07:49,582 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:07:49,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:52,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:07:55,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:07:56,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:00,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:04,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:08:04,582 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:08:05,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:06,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:10,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:14,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:16,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:20,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:08:20,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:08:20,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:08:22,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:22,675 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:08:23,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:27,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:31,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:33,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:35,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:08:35,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:08:35,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:08:38,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:41,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:43,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:49,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:08:49,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:50,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:08:50,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:08:52,676 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:08:55,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:08:59,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:00,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:05,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:09:05,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:09:05,766 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:09:06,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:09,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:11,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:16,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:09:16,456 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:09:16,456 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:09:16,458 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:09:16,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:09:17,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:17,698 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:18,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:20,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:09:20,763 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:09:22,677 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:09:22,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:27,920 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:28,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:33,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:35,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:09:35,764 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:09:37,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:38,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:43,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:46,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:49,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:50,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:09:50,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:09:50,766 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:09:52,678 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:09:54,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:09:56,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:09:59,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:04,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:04,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:05,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:10:05,764 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:10:09,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:14,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:15,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:20,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:10:20,766 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:10:20,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:22,678 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:10:24,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:26,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:31,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:32,318 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:35,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:10:35,765 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:10:37,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:42,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:42,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:47,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:50,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:10:50,765 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:10:51,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:10:52,689 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:10:52,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:57,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:10:59,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:03,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:06,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:11:06,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:11:06,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:11:08,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:09,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:13,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:17,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:18,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:21,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:11:21,204 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:11:22,679 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:11:24,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:27,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:30,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:35,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:36,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:11:36,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:11:37,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:40,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:45,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:46,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:51,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:11:52,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:11:52,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:11:52,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:11:52,682 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:11:55,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:11:56,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:01,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:03,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:06,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:07,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:12:07,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:12:11,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:14,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:16,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:20,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:12:20,885 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:12:20,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:12:20,887 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:12:21,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:12:21,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:22,105 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:22,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:12:22,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:12:22,682 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:12:23,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:27,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:32,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:33,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:38,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:38,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:12:38,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:12:38,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:12:42,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:43,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:48,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:50,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:12:52,682 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:12:53,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:12:53,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:12:53,169 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:12:54,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:12:59,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:00,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:04,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:08,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:13:08,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:13:08,169 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:13:10,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:10,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:15,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:18,551 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:21,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:22,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:13:23,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:13:23,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:13:23,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:13:26,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:27,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:32,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:36,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:37,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:38,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:13:38,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:13:38,964 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:13:42,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:43,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:48,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:52,683 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:13:53,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:13:53,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:13:53,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:13:53,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:13:53,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:13:59,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:03,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:04,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:09,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:09,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:14:09,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:14:09,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:14:11,914 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:14,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:20,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:22,097 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:22,684 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:14:24,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:14:24,568 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:14:25,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:30,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:30,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:35,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:39,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:14:39,568 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:14:40,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:41,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:47,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:48,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:14:52,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:52,686 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:14:54,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:14:54,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:14:54,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:14:57,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:14:58,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:02,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:08,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:08,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:09,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:15:09,573 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:15:13,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:16,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:19,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:22,687 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:15:23,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:15:23,593 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:15:23,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:15:23,595 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:15:24,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:15:24,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:15:24,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:15:24,575 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:15:24,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:25,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:26,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:29,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:34,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:35,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:40,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:40,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:15:40,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:15:40,910 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:15:44,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:46,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:51,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:15:52,687 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:15:52,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:15:55,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:15:55,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:15:55,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:15:56,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:01,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:02,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:06,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:10,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:16:10,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:16:10,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:16:12,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:12,843 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:17,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:20,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:22,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:22,687 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:16:26,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:16:26,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:16:27,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:29,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:32,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:38,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:38,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:41,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:16:41,505 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:16:43,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:48,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:49,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:52,689 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:16:54,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:16:56,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:16:56,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:16:56,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:17:00,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:05,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:06,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:10,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:12,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:17:12,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:17:12,826 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:17:15,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:16,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:20,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:22,700 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:17:26,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:27,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:17:27,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:17:27,826 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:17:30,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:32,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:37,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:40,447 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:42,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:42,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:17:42,824 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:17:48,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:48,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:52,689 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:17:53,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:17:57,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:17:57,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:17:58,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:17:59,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:04,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:06,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:09,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:13,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:18:13,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:18:13,159 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:18:15,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:16,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:20,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:22,691 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:18:24,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:25,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:28,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:18:28,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:18:28,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:18:30,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:33,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:18:33,193 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:18:33,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:18:33,197 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:18:33,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:18:34,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:35,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:36,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:41,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:43,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:18:43,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:18:43,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:18:46,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:51,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:52,692 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:18:56,955 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:18:57,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:18:58,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:18:58,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:18:58,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:19:02,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:07,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:07,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:12,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:13,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:19:13,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:19:13,159 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:19:17,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:18,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:22,693 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:19:23,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:25,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:28,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:19:28,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:19:28,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:19:29,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:34,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:35,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:40,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:42,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:43,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:19:43,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:19:43,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:19:45,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:50,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:52,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:19:52,693 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:19:55,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:19:58,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:19:58,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:19:58,814 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:20:00,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:02,369 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:05,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:10,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:10,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:13,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:20:13,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:20:13,814 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:20:16,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:20,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:21,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:22,694 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:20:26,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:28,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:28,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:20:28,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:20:28,814 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:20:32,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:37,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:38,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:42,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:44,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:20:44,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:20:44,769 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:20:47,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:48,693 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:52,696 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:20:53,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:56,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:20:58,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:20:59,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:20:59,767 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:21:04,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:06,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:09,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:14,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:14,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:21:14,767 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:21:14,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:20,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:22,696 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:21:24,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:25,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:30,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:31,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:21:31,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:21:31,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:21:35,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:36,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:41,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:43,080 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:46,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:21:46,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:21:46,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:50,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:21:50,346 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:21:50,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:21:50,349 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:21:51,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:21:52,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:52,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:21:52,697 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:21:53,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:21:57,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:01,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:01,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:22:01,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:22:03,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:08,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:11,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:13,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:16,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:22:16,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:22:16,960 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:22:19,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:21,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:22,697 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:22:24,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:28,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:30,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:31,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:22:31,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:22:31,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:22:35,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:38,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:41,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:46,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:46,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:22:46,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:22:46,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:22:46,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:22:51,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:52,698 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:22:56,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:22:56,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:01,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:02,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:23:02,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:23:02,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:23:04,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:06,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:11,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:14,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:16,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:17,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:23:17,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:23:17,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:23:21,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:22,698 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:23:22,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:27,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:32,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:23:32,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:23:32,496 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:23:32,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:32,843 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:37,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:40,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:42,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:47,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:23:47,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:23:47,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:23:48,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:50,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:23:52,709 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:23:53,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:23:59,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:01,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:02,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:24:02,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:24:02,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:24:05,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:09,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:10,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:15,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:17,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:24:17,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:24:17,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:24:19,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:20,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:22,700 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:24:26,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:27,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:31,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:33,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:24:33,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:24:33,131 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:24:36,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:37,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:41,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:45,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:46,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:48,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:24:48,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:24:48,130 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:24:52,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:24:52,701 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:24:52,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:24:52,866 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:24:52,866 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:24:52,869 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:24:53,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:24:54,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:55,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:24:57,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:03,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:25:03,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:25:03,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:03,131 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:25:03,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:08,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:13,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:13,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:18,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:25:18,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:25:18,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:25:18,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:20,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:22,701 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:25:24,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:29,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:30,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:33,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:25:33,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:25:34,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:38,731 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:40,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:45,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:48,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:25:48,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:25:48,788 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:25:50,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:52,701 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:25:56,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:25:58,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:01,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:04,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:26:04,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:26:04,152 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:26:06,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:07,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:12,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:14,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:17,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:19,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:26:19,150 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:26:22,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:22,705 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:26:25,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:28,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:33,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:34,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:26:34,150 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:26:34,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:39,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:43,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:45,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:50,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:26:50,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:26:50,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:26:50,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:26:52,705 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:26:53,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:26:55,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:01,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:01,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:05,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:27:05,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:27:05,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:27:06,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:11,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:12,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:17,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:20,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:27:20,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:27:20,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:27:21,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:22,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:22,705 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:27:27,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:29,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:33,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:36,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:27:36,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:27:36,435 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:27:38,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:39,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:43,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:48,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:49,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:51,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:27:51,432 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:27:52,706 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:27:54,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:27:55,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:27:55,927 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:27:55,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:27:55,929 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:27:56,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:27:56,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:58,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:27:59,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:05,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:06,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:28:06,432 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:28:06,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:11,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:14,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:16,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:21,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:28:21,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:28:21,614 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:28:21,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:22,707 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:28:25,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:27,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:33,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:35,275 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:36,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:28:36,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:28:38,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:43,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:44,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:49,015 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:51,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:28:51,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:28:52,707 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:28:53,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:28:54,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:28:59,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:01,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:05,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:07,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:29:07,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:29:07,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:29:10,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:11,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:15,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:19,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:21,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:22,708 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:29:22,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:29:22,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:29:22,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:29:27,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:29,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:32,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:37,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:29:37,844 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:29:38,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:39,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:43,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:48,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:49,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:29:52,708 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:29:52,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:29:52,844 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:29:53,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:59,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:29:59,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:04,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:07,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:30:07,844 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:30:07,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:10,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:15,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:18,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:20,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:22,710 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:30:23,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:30:23,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:30:23,128 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:30:25,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:25,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:30,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:35,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:35,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:38,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:30:38,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:30:38,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:30:41,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:45,222 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:46,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:51,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:30:52,710 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:30:53,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:30:53,125 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:30:55,294 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:30:57,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:02,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:02,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:31:02,315 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:31:02,316 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:31:02,318 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:31:02,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:31:03,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:31:04,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:31:07,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:08,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:31:08,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:31:08,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:31:12,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:17,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:22,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:22,710 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:31:23,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:31:23,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:31:28,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:33,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:38,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:31:38,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:31:38,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:43,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:48,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:52,711 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:31:53,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:31:53,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:31:54,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:31:59,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:03,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:05,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:09,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:32:09,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:32:09,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:32:10,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:11,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:15,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:20,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:21,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:22,711 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:32:24,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:32:24,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:32:26,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:29,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:31,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:36,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:39,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:32:39,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:32:39,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:42,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:47,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:50,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:32:52,543 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:52,712 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:32:55,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:32:55,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:32:55,630 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:32:57,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:32:58,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:02,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:08,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:08,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:10,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:33:10,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:33:13,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:16,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:19,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:22,713 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:33:24,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:25,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:33:25,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:33:26,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:29,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:35,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:35,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:40,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:41,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:33:41,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:33:41,579 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:33:43,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:45,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:50,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:52,713 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:33:53,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:33:56,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:33:56,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:33:56,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:33:56,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:34:02,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:03,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:07,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:11,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:11,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:34:11,578 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:34:12,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:17,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:21,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:22,713 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:34:23,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:27,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:34:27,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:34:27,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:34:29,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:29,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:34,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:39,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:39,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:42,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:34:42,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:34:42,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:34:45,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:47,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:50,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:52,714 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:34:55,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:34:55,725 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:34:55,726 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:34:55,728 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:34:55,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:34:56,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:34:56,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:34:57,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:34:57,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:34:57,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:34:57,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:02,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:05,947 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:07,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:12,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:12,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:35:12,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:35:12,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:35:16,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:18,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:22,716 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:35:23,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:26,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:27,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:35:27,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:35:27,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:35:29,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:34,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:34,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:39,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:42,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:35:42,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:35:42,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:35:44,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:44,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:49,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:52,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:35:52,716 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:35:55,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:35:58,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:35:58,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:35:58,735 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:36:00,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:02,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:05,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:10,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:11,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:13,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:36:13,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:36:16,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:20,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:22,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:22,716 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:36:28,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:28,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:36:28,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:36:29,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:33,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:37,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:39,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:43,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:36:43,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:36:43,967 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:36:45,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:47,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:50,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:52,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:36:55,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:36:56,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:36:58,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:36:58,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:37:01,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:05,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:06,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:11,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:13,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:13,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:37:13,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:37:17,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:22,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:37:22,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:23,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:27,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:29,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:37:29,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:37:29,500 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:37:32,001 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:33,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:38,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:42,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:43,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:44,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:37:44,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:37:49,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:50,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:52,718 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:37:54,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:37:58,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:37:58,471 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:37:58,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:37:58,473 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:37:59,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:37:59,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:37:59,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:37:59,497 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:38:00,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:00,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:05,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:10,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:10,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:15,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:38:15,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:38:15,660 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:38:15,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:18,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:20,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:22,719 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:38:26,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:28,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:30,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:38:30,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:38:30,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:38:31,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:36,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:37,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:42,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:45,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:38:45,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:38:45,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:38:46,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:47,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:52,720 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:38:53,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:38:56,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:38:58,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:00,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:39:00,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:39:00,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:39:04,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:06,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:09,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:14,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:15,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:39:15,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:39:15,659 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:39:15,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:19,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:22,720 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:39:23,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:25,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:30,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:31,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:39:31,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:39:31,073 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:39:33,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:36,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:41,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:43,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:46,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:39:46,071 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:39:46,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:51,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:39:52,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:39:52,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:39:56,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:01,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:40:01,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:40:01,072 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:40:02,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:02,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:07,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:10,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:13,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:17,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:40:17,295 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:40:18,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:20,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:22,721 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:40:23,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:28,277 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:29,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:32,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:40:32,294 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:40:34,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:38,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:40,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:45,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:47,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:40:47,295 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:40:48,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:40:50,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:52,724 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:40:55,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:40:56,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:00,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:02,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:41:02,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:41:02,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:41:04,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:41:04,416 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:41:04,417 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:41:04,419 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:41:04,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:41:05,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:06,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:06,535 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:11,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:16,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:16,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:17,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:41:17,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:41:17,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:41:22,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:22,723 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:41:24,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:27,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:32,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:41:32,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:41:33,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:34,715 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:38,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:43,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:44,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:47,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:41:47,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:41:47,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:41:49,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:41:52,723 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:41:52,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:41:54,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:00,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:02,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:42:02,921 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:02,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:42:06,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:09,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:11,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:16,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:17,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:42:17,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:42:17,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:42:20,095 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:22,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:22,724 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:42:27,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:30,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:32,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:32,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:42:32,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:42:38,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:38,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:43,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:47,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:42:47,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:42:48,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:48,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:52,726 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:42:53,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:42:56,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:42:58,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:02,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:43:02,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:43:02,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:43:04,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:06,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:09,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:14,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:16,466 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:17,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:43:17,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:43:17,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:43:20,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:22,737 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:43:24,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:25,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:30,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:32,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:43:32,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:43:34,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:36,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:41,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:42,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:46,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:47,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:43:47,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:43:47,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:43:52,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:43:52,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:43:52,726 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:43:57,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:02,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:02,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:02,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:44:02,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:44:08,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:09,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:44:09,049 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:44:09,050 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:44:09,051 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:44:09,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:44:10,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:11,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:14,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:17,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:44:18,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:44:18,026 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:44:19,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:20,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:22,727 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:44:24,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:28,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:30,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:32,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:44:32,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:44:36,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:39,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:41,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:46,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:47,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:44:47,969 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:44:49,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:52,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:44:52,729 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:44:56,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:44:57,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:02,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:03,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:45:03,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:45:03,498 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:45:06,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:45:07,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:12,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:14,273 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:45:18,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:18,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:45:18,494 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:45:22,728 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:45:23,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:28,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:30,372 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:45:33,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:45:33,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:45:33,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:45:34,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:38,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:45:40,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:45,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:48,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:45:48,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:45:48,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:45:50,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:52,731 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:45:55,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:45:56,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:00,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:03,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:46:03,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:46:03,873 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:46:05,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:06,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:10,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:16,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:16,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:18,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:46:18,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:46:18,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:46:22,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:22,730 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:46:24,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:27,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:32,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:33,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:46:33,872 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:46:34,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:38,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:43,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:44,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:48,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:48,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:46:48,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:46:48,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:46:52,731 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:46:52,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:46:53,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:46:58,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:03,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:47:03,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:47:03,873 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:47:04,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:09,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:14,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:18,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:47:18,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:47:20,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:22,731 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:47:23,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:47:25,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:29,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:47:29,765 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:47:29,766 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:47:29,768 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:47:30,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:47:30,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:31,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:47:33,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:47:33,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:47:36,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:41,311 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:47:41,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:46,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:48,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:47:48,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:47:48,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:47:52,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:52,732 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:47:57,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:47:58,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:02,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:05,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:48:05,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:48:05,330 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:48:08,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:08,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:13,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:16,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:18,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:20,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:48:20,328 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:48:22,733 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:48:23,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:26,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:29,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:34,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:34,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:35,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:48:35,328 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:48:40,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:44,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:46,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:50,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:48:50,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:48:50,937 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:48:51,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:48:52,734 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:48:52,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:48:56,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:02,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:02,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:05,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:49:05,934 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:49:08,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:12,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:13,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:18,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:20,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:49:20,934 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:49:21,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:22,734 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:49:23,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:29,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:31,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:34,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:36,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:49:36,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:49:36,393 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:49:39,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:40,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:45,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:49,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:49:50,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:51,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:49:51,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:49:52,736 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:49:55,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:49:57,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:00,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:06,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:06,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:50:06,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:50:07,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:11,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:15,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:16,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:21,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:50:21,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:50:21,447 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:50:21,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:22,735 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:50:25,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:26,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:31,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:31,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:50:31,866 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:50:31,866 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:50:31,868 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:50:32,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:50:33,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:36,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:50:36,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:50:36,446 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:50:37,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:43,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:43,581 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:48,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:50,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:50:51,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:50:51,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:50:51,446 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:50:52,736 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:50:53,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:50:58,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:00,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:51:04,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:06,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:51:06,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:51:06,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:51:08,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:51:09,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:14,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:18,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:51:20,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:21,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:51:21,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:51:21,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:51:22,736 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:51:25,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:28,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:51:30,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:35,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:36,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:51:36,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:51:36,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:51:41,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:46,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:51,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:51:51,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:51:51,876 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:51:52,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:51:52,737 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:51:57,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:03,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:03,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:06,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:52:06,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:52:06,826 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:52:09,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:11,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:14,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:19,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:21,233 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:21,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:52:21,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:52:21,827 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:52:22,738 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:52:24,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:30,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:31,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:35,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:36,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:52:36,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:52:36,830 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:52:39,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:41,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:46,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:49,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:52:51,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:51,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:52:51,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:52:51,827 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:52:52,738 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:52:57,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:52:59,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:02,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:06,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:53:06,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:53:06,828 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:53:07,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:08,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:13,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:15,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:18,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:22,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:53:22,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:53:22,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:53:22,738 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:53:24,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:25,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:29,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:34,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:35,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:37,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:53:37,034 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:53:40,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:43,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:46,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:51,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:52,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:53:52,034 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:53:52,739 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:53:53,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:53:57,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:53:59,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:53:59,796 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:53:59,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:53:59,798 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:53:59,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:54:00,896 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:01,902 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:02,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:07,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:54:07,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:54:07,478 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:54:08,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:08,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:13,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:19,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:19,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:54:22,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:54:22,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:54:22,739 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:54:24,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:29,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:30,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:35,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:37,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:37,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:54:37,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:54:40,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:46,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:47,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:51,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:54:52,741 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:54:53,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:54:53,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:54:53,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:54:55,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:54:56,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:01,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:05,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:07,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:08,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:55:08,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:55:08,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:55:12,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:15,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:18,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:22,741 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:55:23,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:55:23,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:55:23,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:55:24,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:25,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:29,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:33,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:35,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:38,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:55:39,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:55:39,998 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:55:40,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:43,597 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:45,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:51,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:55:51,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:55:52,743 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:55:53,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:55:54,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:55:54,997 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:55:57,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:01,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:03,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:08,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:09,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:09,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:56:09,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:56:13,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:18,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:19,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:22,744 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:56:23,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:25,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:56:25,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:56:27,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:28,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:33,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:37,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:39,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:40,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:56:40,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:56:44,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:48,007 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:56:49,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:52,745 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:56:55,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:56:55,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:56:55,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:56:56,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:00,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:03,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 02:57:03,087 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 02:57:03,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 02:57:03,092 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 02:57:03,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 02:57:04,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:05,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:06,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:10,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:57:10,953 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:57:11,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:13,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:16,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:22,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:22,745 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:57:23,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:25,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:57:25,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:57:28,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:33,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:33,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:38,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:39,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:57:40,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:57:40,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:57:41,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:44,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:49,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:51,513 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:57:52,745 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:57:54,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:57:54,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:57:57,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:57:57,314 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:57:59,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:00,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:05,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:09,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:58:09,639 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:10,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:12,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:58:12,312 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:58:16,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:19,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:22,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:22,746 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:58:25,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:58:27,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:58:27,313 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:58:27,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:27,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:32,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:37,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:37,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:40,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:58:43,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:58:43,507 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:58:43,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:45,921 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:49,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:52,749 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:58:55,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:58:55,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:58:55,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:58:58,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:58:58,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:59:00,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:04,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:06,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:11,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:59:12,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:13,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:59:13,506 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:59:14,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:17,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:22,748 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:59:22,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:24,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:26,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:59:28,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:28,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:59:28,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:59:32,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:33,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:38,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:42,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 02:59:43,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:43,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:59:43,979 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 02:59:45,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:49,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:52,748 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 02:59:53,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 02:59:54,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 02:59:58,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 02:59:58,981 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:00:00,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:03,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:05,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:10,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:11,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:00:11,377 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:00:11,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:00:11,380 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:00:11,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:00:12,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:13,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:13,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:00:13,980 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:00:16,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:21,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:21,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:22,749 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:00:26,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:28,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:00:28,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:00:28,983 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:00:31,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:32,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:37,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:41,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:43,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:43,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:00:43,981 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:00:49,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:51,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:00:52,750 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:00:54,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:00:58,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:00:58,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:00:58,982 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:00:59,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:00,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:05,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:09,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:10,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:13,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:01:13,983 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:01:16,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:19,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:21,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:22,750 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:01:27,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:27,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:28,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:01:28,982 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:01:33,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:38,063 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:38,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:43,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:01:43,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:01:43,983 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:01:44,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:46,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:49,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:52,751 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:01:54,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:01:56,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:01:58,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:01:58,982 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:02:00,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:05,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:06,248 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:10,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:13,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:02:13,983 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:02:14,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:16,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:21,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:22,751 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:02:24,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:26,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:29,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:02:29,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:02:29,933 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:02:31,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:32,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:36,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:42,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:42,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:44,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:02:44,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:02:48,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:51,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:52,762 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:02:53,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:58,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:02:59,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:02:59,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:02:59,930 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:03:04,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:09,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:09,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:14,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:16,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:03:16,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:03:16,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:03:17,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:03:17,486 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:03:17,486 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:03:17,488 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:03:17,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:03:18,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:19,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:20,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:22,752 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:03:25,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:27,817 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:31,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:03:31,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:03:31,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:31,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:03:36,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:37,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:41,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:45,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:46,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:03:46,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:03:46,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:03:47,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:52,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:03:52,754 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:03:56,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:03:57,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:01,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:04:01,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:04:01,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:04:03,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:04,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:08,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:14,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:14,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:16,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:04:16,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:04:19,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:22,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:22,754 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:04:25,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:30,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:31,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:04:31,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:04:32,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:35,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:40,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:41,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:46,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:04:46,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:04:46,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:04:47,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:50,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:52,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:04:52,754 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:04:58,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:04:58,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:01,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:05:01,755 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:05:04,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:04,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:10,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:10,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:15,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:16,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:05:16,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:05:16,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:05:20,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:20,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:22,755 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:05:25,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:28,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:31,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:31,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:05:31,755 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:05:36,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:38,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:41,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:46,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:05:46,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:05:46,779 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:47,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:52,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:05:53,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:05:55,840 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:05:59,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:02,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:06:02,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:06:02,024 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:06:03,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:05,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:10,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:12,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:06:12,374 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:06:12,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:06:12,378 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:06:12,948 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:06:13,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:14,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:15,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:17,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:06:17,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:06:17,022 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:06:21,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:22,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:22,757 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:06:26,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:32,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:06:32,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:32,022 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:06:32,080 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:37,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:42,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:42,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:47,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:48,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:06:48,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:06:48,005 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:06:50,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:06:52,757 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:06:53,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:06:58,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:00,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:03,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:07:03,004 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:07:04,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:08,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:09,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:14,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:18,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:07:18,004 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:07:18,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:20,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:22,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:07:25,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:26,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:30,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:33,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:07:33,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:07:33,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:07:36,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:36,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:41,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:44,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:47,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:48,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:07:48,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:07:48,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:07:52,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:07:52,758 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:07:54,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:07:57,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:03,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:08:03,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:08:03,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:03,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:08:04,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:08,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:12,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:13,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:18,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:18,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:08:18,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:08:18,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:08:22,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:22,759 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:08:23,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:29,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:30,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:33,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:08:33,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:08:33,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:08:34,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:38,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:40,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:45,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:48,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:48,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:08:48,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:08:48,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:08:50,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:08:52,762 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:08:56,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:08:56,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:01,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:03,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:09:03,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:09:03,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:09:06,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:07,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:12,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:13,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:09:13,904 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:09:13,905 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:09:13,907 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:09:14,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:09:15,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:16,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:17,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:18,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:09:18,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:09:18,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:09:22,763 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:09:23,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:24,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:28,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:33,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:09:33,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:09:33,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:09:33,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:34,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:38,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:44,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:44,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:49,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:50,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:09:50,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:09:50,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:09:52,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:09:52,763 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:09:54,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:09:59,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:02,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:04,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:05,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:10:05,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:10:05,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:10:10,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:10,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:15,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:20,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:10:20,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:10:20,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:21,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:22,764 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:10:26,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:28,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:31,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:35,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:10:35,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:10:35,599 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:10:37,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:39,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:42,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:47,082 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:48,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:50,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:10:50,598 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:10:52,766 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:10:53,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:10:57,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:10:58,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:03,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:05,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:11:05,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:11:05,598 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:11:09,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:14,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:19,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:21,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:11:21,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:11:21,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:11:22,766 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:11:24,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:11:25,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:30,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:34,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:11:36,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:36,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:11:36,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:11:41,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:42,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:11:46,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:51,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:11:51,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:11:51,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:11:52,505 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:11:52,766 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:11:56,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:00,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:02,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:07,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:12:07,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:12:07,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:12:07,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:10,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:13,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:18,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:19,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:22,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:12:22,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:12:22,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:12:22,767 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:12:24,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:26,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:12:26,728 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:12:26,728 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:12:26,731 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:12:26,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:12:27,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:28,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:30,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:35,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:36,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:37,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:12:37,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:12:41,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:46,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:46,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:51,672 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:12:52,768 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:12:52,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:12:52,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:12:52,815 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:12:54,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:12:56,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:01,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:05,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:07,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:07,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:13:07,813 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:13:13,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:15,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:18,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:22,769 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:13:22,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:13:22,813 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:13:24,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:29,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:29,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:34,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:37,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:37,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:13:37,813 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:13:39,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:45,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:47,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:50,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:13:52,771 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:13:52,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:13:52,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:13:52,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:13:55,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:13:55,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:00,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:05,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:06,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:07,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:14:07,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:14:12,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:12,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:17,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:22,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:22,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:22,770 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:14:22,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:14:22,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:14:28,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:30,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:33,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:38,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:14:38,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:14:38,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:14:38,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:40,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:44,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:48,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:14:49,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:52,771 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:14:53,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:14:53,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:14:55,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:14:58,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:00,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:06,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:06,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:08,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:15:08,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:15:12,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:16,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:17,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:22,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:22,772 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:15:23,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:15:23,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:15:23,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:15:26,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:27,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:32,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:34,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:15:34,260 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:15:34,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:15:34,262 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:15:35,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:15:35,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:36,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:38,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:38,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:15:38,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:15:38,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:15:43,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:45,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:48,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:52,773 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:15:53,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:15:53,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:15:53,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:15:53,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:15:55,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:15:58,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:03,177 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:04,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:08,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:16:08,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:16:09,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:13,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:15,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:20,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:22,773 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:16:23,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:23,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:16:23,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:16:25,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:31,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:31,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:36,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:38,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:16:38,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:16:38,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:16:41,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:41,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:46,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:49,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:16:52,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:52,774 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:16:53,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:16:53,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:16:58,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:16:59,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:03,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:08,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:08,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:17:08,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:17:11,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:13,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:18,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:19,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:22,774 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:17:23,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:17:23,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:17:23,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:29,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:29,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:34,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:36,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:38,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:17:38,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:17:39,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:44,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:46,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:50,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:17:52,777 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:17:53,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:17:53,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:17:54,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:17:55,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:00,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:04,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:05,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:08,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:18:08,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:18:11,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:13,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:17,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:22,777 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:18:22,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:23,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:23,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:18:23,340 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:18:28,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:33,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:34,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:39,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:39,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:18:39,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:18:39,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:18:40,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:18:40,373 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:18:40,374 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:18:40,376 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:18:41,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:18:41,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:42,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:44,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:49,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:18:51,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:18:52,777 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:18:54,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:18:54,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:18:55,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:00,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:01,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:05,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:09,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:19:09,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:19:09,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:11,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:16,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:19,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:22,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:22,777 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:19:25,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:19:25,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:19:25,161 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:19:27,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:27,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:32,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:37,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:37,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:40,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:19:40,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:19:40,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:19:43,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:45,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:19:48,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:52,779 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:19:54,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:19:55,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:19:55,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:19:55,159 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:19:55,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:00,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:03,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:05,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:10,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:20:10,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:20:10,414 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:20:10,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:13,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:16,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:21,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:21,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:22,779 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:20:25,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:20:25,412 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:20:27,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:30,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:33,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:38,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:40,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:20:40,413 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:20:41,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:43,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:48,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:51,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:52,779 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:20:53,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:20:55,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:20:55,415 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:20:59,166 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:20:59,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:04,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:09,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:09,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:10,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:21:10,413 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:21:15,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:17,277 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:21,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:22,780 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:21:25,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:21:25,414 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:21:26,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:27,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:31,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:37,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:37,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:40,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:21:40,413 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:21:42,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:43,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:21:43,859 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:21:43,859 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:21:43,862 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:21:44,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:21:45,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:47,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:52,780 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:21:52,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:21:53,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:21:55,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:21:55,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:21:55,420 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:21:58,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:03,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:03,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:08,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:10,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:22:10,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:22:10,422 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:22:13,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:14,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:19,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:21,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:22,781 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:22:24,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:25,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:22:25,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:22:25,421 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:22:29,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:31,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:34,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:39,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:40,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:22:40,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:40,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:22:40,422 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:22:45,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:49,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:22:50,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:52,781 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:22:55,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:22:56,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:22:56,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:22:56,045 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:22:57,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:01,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:06,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:08,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:11,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:23:11,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:23:11,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:23:12,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:17,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:18,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:22,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:22,783 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:23:26,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:23:26,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:23:26,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:23:26,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:28,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:33,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:36,203 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:38,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:41,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:23:41,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:23:41,478 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:23:43,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:44,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:48,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:52,783 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:23:53,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:23:54,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:23:56,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:23:56,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:23:59,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:03,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:05,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:10,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:11,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:24:11,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:24:11,655 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:15,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:20,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:21,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:22,783 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:24:26,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:27,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:24:27,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:24:27,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:24:29,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:31,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:36,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:39,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:42,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:24:42,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:42,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:24:47,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:24:47,157 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:24:47,157 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:24:47,159 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:24:47,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:24:47,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:48,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:52,784 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:24:53,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:24:57,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:24:57,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:24:57,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:24:59,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:04,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:06,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:25:09,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:12,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:25:12,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:25:12,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:25:14,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:16,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:25:19,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:22,784 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:25:24,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:25:24,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:27,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:25:27,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:25:29,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:34,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:39,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:42,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:25:42,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:25:45,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:50,313 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:25:51,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:52,785 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:25:56,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:25:57,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:25:57,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:26:00,385 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:02,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:07,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:10,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:12,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:12,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:26:12,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:26:17,512 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:18,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:22,785 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:26:23,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:27,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:27,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:26:27,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:26:29,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:34,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:35,637 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:39,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:42,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:26:42,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:26:42,867 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:26:44,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:45,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:49,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:52,797 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:26:53,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:26:55,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:26:57,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:26:57,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:26:57,866 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:27:01,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:03,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:06,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:11,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:14,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:27:14,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:27:14,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:27:17,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:17,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:22,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:22,799 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:27:28,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:28,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:29,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:27:29,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:27:34,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:36,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:39,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:44,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:27:44,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:27:44,287 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:27:45,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:46,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:50,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:52,800 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:27:55,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:27:56,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:27:59,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:27:59,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:28:00,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:04,246 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:06,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:11,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:28:11,219 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:28:11,220 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:28:11,222 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:28:11,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:28:12,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:12,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:14,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:28:14,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:28:17,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:22,371 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:22,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:22,800 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:28:27,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:29,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:28:29,287 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:28:32,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:33,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:39,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:40,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:44,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:28:44,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:28:45,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:50,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:50,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:52,800 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:28:55,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:28:58,594 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:28:59,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:28:59,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:29:01,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:06,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:08,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:11,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:14,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:29:14,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:29:14,288 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:29:16,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:16,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:21,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:22,802 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:29:25,770 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:27,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:29,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:29:29,287 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:29:32,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:33,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:38,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:43,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:43,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:44,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:29:44,290 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:29:48,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:52,818 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:29:53,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:53,968 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:29:58,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:29:59,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:29:59,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:29:59,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:30:02,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:03,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:08,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:12,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:14,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:14,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:30:14,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:30:20,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:20,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:22,803 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:30:25,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:29,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:30:29,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:30:30,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:31,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:36,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:38,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:41,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:45,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:30:45,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:30:45,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:30:47,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:48,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:52,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:30:52,805 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:30:56,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:30:57,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:00,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:31:00,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:31:00,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:31:02,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:06,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:08,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:13,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:14,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:31:14,337 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:31:14,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:31:14,340 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:31:14,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:31:15,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:31:15,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:31:15,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:31:16,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:18,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:22,805 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:31:24,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:24,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:29,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:31,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:31:31,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:31:31,278 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:31:34,686 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:35,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:40,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:44,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:45,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:46,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:31:46,276 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:31:51,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:31:52,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:31:52,806 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:31:56,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:01,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:32:01,276 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:32:02,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:02,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:07,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:10,940 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:13,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:17,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:32:17,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:32:17,148 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:32:18,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:19,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:22,806 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:32:23,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:28,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:29,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:32,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:32:32,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:32:35,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:38,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:40,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:45,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:47,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:32:47,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:32:48,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:51,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:32:52,808 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:32:56,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:32:57,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:02,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:03,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:33:03,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:33:03,128 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:33:06,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:08,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:13,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:14,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:18,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:33:18,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:33:19,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:22,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:22,808 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:33:24,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:29,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:32,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:33,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:33:33,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:33:35,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:41,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:42,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:46,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:48,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:33:48,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:33:48,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:33:50,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:33:51,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:33:52,808 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:33:56,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:00,678 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:02,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:03,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:34:03,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:34:03,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:34:07,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:08,737 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:12,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:16,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:34:16,814 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:34:16,814 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:34:16,816 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:34:17,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:34:17,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:18,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:34:18,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:34:18,444 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:34:18,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:22,823 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:34:22,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:26,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:28,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:33,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:34:33,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:34:33,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:34:33,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:36,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:39,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:44,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:44,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:48,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:34:48,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:34:48,548 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:34:50,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:34:52,810 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:34:55,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:34:56,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:01,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:03,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:35:03,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:35:03,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:35:05,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:06,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:12,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:12,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:17,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:19,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:35:19,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:35:19,741 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:35:22,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:22,811 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:35:22,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:27,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:30,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:33,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:34,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:35:34,740 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:35:38,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:40,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:44,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:49,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:49,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:35:49,740 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:35:50,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:35:52,811 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:35:54,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:35:58,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:00,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:05,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:36:05,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:36:05,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:36:06,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:08,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:11,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:16,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:17,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:20,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:36:20,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:36:20,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:36:22,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:22,812 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:36:26,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:27,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:32,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:35,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:36:35,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:36:35,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:36:36,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:38,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:44,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:44,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:49,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:51,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:36:51,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:36:51,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:36:52,813 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:36:54,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:36:55,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:36:59,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:03,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:05,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:06,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:37:06,538 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:37:11,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:13,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:16,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:19,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:37:19,553 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:37:19,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:37:19,556 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:37:20,261 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:37:21,266 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:21,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:37:21,538 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:37:21,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:22,814 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:37:26,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:31,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:32,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:37,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:37:37,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:37:37,287 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:37:37,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:40,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:43,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:48,460 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:37:49,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:52,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:37:52,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:37:52,291 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:37:52,814 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:37:54,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:37:58,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:00,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:06,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:06,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:07,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:38:07,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:38:07,285 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:38:11,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:14,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:17,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:22,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:22,815 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:38:22,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:38:22,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:38:22,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:38:24,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:28,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:33,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:34,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:37,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:38:37,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:38:37,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:38:38,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:42,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:44,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:49,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:52,816 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:38:52,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:38:52,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:38:53,010 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:38:53,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:38:54,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:38:59,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:01,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:05,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:08,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:39:08,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:39:08,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:39:10,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:11,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:15,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:19,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:21,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:22,816 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:39:23,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:39:23,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:39:26,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:29,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:31,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:37,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:38,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:39:38,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:39:39,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:43,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:46,585 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:39:49,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:52,817 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:39:54,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:39:54,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:39:54,249 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:39:54,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:39:56,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:00,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:05,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:06,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:09,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:40:09,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:40:09,248 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:40:11,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:14,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:16,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:21,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:22,817 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:40:22,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:40:22,993 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:40:22,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:40:22,996 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:40:23,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:40:24,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:40:24,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:40:24,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:40:24,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:27,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:32,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:37,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:38,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:39,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:40:39,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:40:39,248 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:40:43,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:49,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:49,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:52,828 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:40:54,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:40:54,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:40:54,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:40:54,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:40:57,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:40:59,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:04,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:07,162 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:09,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:41:09,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:41:09,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:41:09,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:15,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:15,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:20,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:22,818 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:41:24,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:41:24,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:41:25,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:26,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:31,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:35,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:36,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:40,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:41:40,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:41:40,262 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:41:42,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:43,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:47,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:52,820 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:41:52,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:41:53,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:41:55,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:41:55,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:41:55,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:41:58,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:01,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:03,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:08,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:10,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:42:10,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:42:11,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:14,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:19,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:20,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:22,821 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:42:25,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:26,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:42:26,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:42:26,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:42:29,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:31,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:36,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:38,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:41,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:42:41,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:42:41,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:42:41,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:46,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:46,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:51,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:42:52,822 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:42:56,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:42:56,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:42:56,171 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:42:56,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:42:57,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:02,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:08,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:09,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:43:11,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:43:11,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:43:11,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:43:13,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:18,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:19,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:43:22,822 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:43:24,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:26,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:43:26,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:43:26,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:43:29,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:43:30,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:35,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:36,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:43:36,966 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:43:36,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:43:36,970 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:43:37,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:43:39,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:43:40,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:41,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:43:41,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:43:41,180 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:43:46,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:49,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:43:51,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:52,823 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:43:56,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:43:56,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:43:56,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:43:57,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:43:59,337 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:02,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:07,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:08,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:11,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:44:11,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:44:11,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:44:14,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:17,456 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:20,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:22,824 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:44:25,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:26,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:44:26,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:44:26,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:44:27,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:31,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:36,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:37,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:41,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:44:41,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:44:41,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:44:42,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:47,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:48,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:52,826 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:44:53,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:44:56,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:44:56,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:44:56,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:44:57,731 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:44:59,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:04,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:07,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:09,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:11,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:45:11,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:45:11,176 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:45:15,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:15,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:20,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:22,826 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:45:25,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:26,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:26,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:45:26,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:45:26,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:45:31,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:35,990 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:37,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:42,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:42,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:45:42,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:45:42,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:45:46,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:47,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:52,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:45:52,835 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:45:55,117 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:45:57,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:45:57,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:45:57,040 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:45:57,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:02,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:03,174 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:07,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:11,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:12,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:46:12,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:46:12,041 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:46:13,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:18,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:21,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:22,837 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:46:23,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:28,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:46:28,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:46:28,043 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:46:29,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:31,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:34,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:39,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:39,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:43,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:46:43,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:46:45,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:47,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:46:47,986 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:46:47,986 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:46:47,988 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:46:48,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:46:49,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:46:50,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:52,837 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:46:56,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:46:58,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:46:58,042 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:46:59,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:02,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:07,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:07,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:12,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:13,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:47:13,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:47:13,045 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:47:17,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:17,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:22,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:22,841 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:47:27,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:28,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:28,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:47:28,043 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:47:33,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:35,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:39,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:43,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:47:43,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:47:43,047 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:47:44,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:45,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:49,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:52,838 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:47:54,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:47:56,001 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:47:58,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:47:58,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:48:00,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:04,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:06,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:11,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:13,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:48:13,044 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:48:14,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:16,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:21,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:22,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:22,839 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:48:26,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:28,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:48:28,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:48:28,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:48:32,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:32,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:37,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:41,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:43,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:43,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:48:43,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:48:48,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:51,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:48:52,839 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:48:54,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:48:58,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:48:58,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:48:58,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:48:59,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:00,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:05,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:09,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:11,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:13,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:49:13,634 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:49:16,411 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:17,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:21,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:22,840 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:49:26,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:27,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:28,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:49:28,632 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:49:31,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:35,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:37,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:42,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:43,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:49:43,634 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:49:45,737 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:48,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:52,842 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:49:53,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:53,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:49:53,355 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:49:53,356 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:49:53,358 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:49:53,793 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:49:53,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:55,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:49:58,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:49:58,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:49:58,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:50:03,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:03,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:08,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:13,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:50:13,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:50:13,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:13,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:18,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:22,841 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:50:23,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:23,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:28,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:29,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:50:29,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:50:29,584 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:50:32,026 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:34,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:39,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:42,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:44,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:50:44,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:50:44,581 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:50:44,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:50,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:52,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:50:52,842 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:50:55,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:50:59,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:50:59,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:50:59,582 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:51:00,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:01,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:06,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:10,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:12,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:15,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:51:15,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:51:15,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:51:17,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:18,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:22,844 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:51:23,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:28,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:28,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:30,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:51:30,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:51:34,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:38,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:39,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:44,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:45,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:51:45,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:51:46,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:51:49,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:52,845 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:51:55,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:51:55,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:00,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:00,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:52:00,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:52:05,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:06,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:11,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:13,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:15,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:52:15,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:52:16,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:21,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:22,845 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:52:23,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:26,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:30,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:52:30,800 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:52:31,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:32,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:37,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:41,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:43,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:45,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:52:45,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:52:48,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:49,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:52:52,846 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:52:54,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:52:58,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:52:58,488 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:52:58,489 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:52:58,492 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:52:59,051 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:53:00,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:00,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:00,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:53:00,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:53:06,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:08,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:11,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:15,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:53:15,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:53:17,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:18,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:22,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:22,847 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:53:26,229 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:28,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:30,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:53:30,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:53:34,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:36,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:39,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:44,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:45,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:53:45,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:53:46,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:49,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:53:52,847 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:53:54,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:53:55,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:00,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:00,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:54:00,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:54:00,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:54:04,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:06,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:11,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:12,565 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:15,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:54:15,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:54:15,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:54:16,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:22,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:22,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:22,848 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:54:27,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:30,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:54:30,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:54:30,951 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:54:32,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:36,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:38,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:43,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:45,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:54:45,948 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:54:46,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:48,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:52,849 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:54:53,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:54:54,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:54:59,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:01,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:55:01,634 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:55:01,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:55:03,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:04,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:09,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:13,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:14,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:16,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:55:16,634 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:55:19,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:22,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:22,849 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:55:25,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:31,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:31,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:55:31,635 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:55:32,083 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:36,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:40,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:42,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:46,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:55:46,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:55:46,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:55:47,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:50,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:55:52,850 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:55:53,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:55:59,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:00,265 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:01,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:56:01,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:56:01,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:56:02,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:04,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:05,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:08,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:56:08,317 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:56:08,318 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:56:08,320 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:56:08,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:56:10,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:10,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:15,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:16,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:56:16,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:56:16,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:56:18,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:21,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:22,850 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:56:26,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:28,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:31,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:31,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:56:31,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:56:31,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:56:36,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:37,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:42,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:46,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:46,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:56:46,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:56:46,906 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:56:48,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:52,851 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:56:53,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:56:54,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:56:58,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:01,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:57:01,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:57:01,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:57:04,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:04,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:09,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:12,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:15,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:16,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:57:16,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:57:16,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:57:21,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:22,836 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:22,851 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:57:26,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:30,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:31,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:57:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:57:31,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:57:37,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:40,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:42,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:46,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:57:46,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:57:46,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:57:48,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:49,010 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:52,852 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:57:53,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:57:59,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:57:59,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:01,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:58:01,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:58:01,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:58:04,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:06,125 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:09,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:15,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:16,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:16,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:58:16,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:58:16,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:58:20,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:22,853 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:58:24,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:25,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:30,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:58:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:58:31,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:58:34,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:36,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:41,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:42,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:46,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:46,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:58:46,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:58:46,910 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:58:50,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:58:52,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:58:52,853 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:58:57,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:00,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:01,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:59:01,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:59:01,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:59:03,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:07,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 03:59:07,807 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 03:59:07,807 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 03:59:07,809 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 03:59:08,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 03:59:08,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:10,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:13,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:16,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:59:16,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:59:16,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:59:18,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:19,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:22,853 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:59:24,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:28,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:30,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:31,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:59:31,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:59:31,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:59:35,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:38,744 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:40,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:46,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:46,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:46,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 03:59:46,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 03:59:46,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 03:59:52,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 03:59:52,855 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 03:59:54,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 03:59:57,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:02,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:00:02,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:00:02,005 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:00:03,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:04,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:08,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:14,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:14,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:17,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:00:17,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:00:17,005 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:00:19,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:22,855 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:00:23,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:24,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:29,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:32,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:00:32,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:00:32,005 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:00:33,089 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:35,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:40,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:41,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:45,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:47,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:00:47,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:00:47,738 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:00:50,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:00:51,234 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:00:52,855 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:00:55,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:00,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:01,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:02,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:01:02,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:01:02,736 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:01:05,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:08,345 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:11,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:16,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:17,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:17,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:01:17,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:01:22,857 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:01:22,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:26,475 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:28,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:32,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:01:32,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:01:32,739 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:01:33,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:34,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:38,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:44,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:44,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:47,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:01:47,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:01:49,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:01:52,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:01:52,857 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:01:55,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:01,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:02,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:02:02,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:02:02,765 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:06,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:09,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:02:09,933 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:02:09,933 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:02:09,936 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:02:10,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:02:12,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:12,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:17,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:02:17,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:02:17,914 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:02:18,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:20,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:22,858 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:02:23,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:28,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:30,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:32,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:02:32,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:02:34,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:39,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:41,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:44,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:47,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:02:47,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:02:49,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:02:50,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:52,858 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:02:55,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:02:59,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:01,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:02,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:03:02,911 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:03:07,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:12,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:17,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:17,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:03:17,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:03:17,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:03:19,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:22,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:22,859 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:03:27,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:29,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:32,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:32,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:03:32,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:03:32,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:03:37,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:37,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:42,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:47,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:47,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:03:47,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:03:47,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:03:48,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:52,860 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:03:53,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:03:55,475 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:03:58,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:02,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:04:02,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:04:02,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:04:03,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:04,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:09,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:14,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:14,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:17,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:04:17,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:04:17,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:04:20,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:22,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:22,860 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:04:25,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:31,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:32,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:32,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:04:32,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:04:32,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:04:37,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:40,778 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:42,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:47,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:47,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:04:47,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:04:47,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:04:50,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:04:52,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:52,861 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:04:58,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:04:58,896 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:02,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:05:02,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:05:02,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:05:04,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:08,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:09,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:14,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:17,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:17,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:05:17,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:05:17,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:05:20,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:22,861 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:05:23,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:05:23,823 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:05:23,824 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:05:23,826 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:05:24,055 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:05:25,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:25,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:30,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:32,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:05:32,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:05:32,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:05:35,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:36,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:41,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:45,177 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:46,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:47,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:05:47,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:05:47,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:05:52,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:05:52,862 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:05:53,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:05:57,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:02,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:06:02,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:06:02,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:06:03,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:03,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:08,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:11,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:13,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:17,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:06:17,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:06:17,918 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:06:19,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:21,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:22,862 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:06:24,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:30,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:31,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:32,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:06:32,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:06:32,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:06:36,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:39,509 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:41,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:47,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:47,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:47,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:06:47,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:06:47,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:06:52,863 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:06:52,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:06:57,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:06:57,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:02,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:07:02,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:07:02,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:07:03,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:05,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:09,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:14,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:15,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:17,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:07:17,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:07:19,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:22,863 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:07:24,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:25,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:29,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:32,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:32,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:07:32,953 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:07:35,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:40,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:42,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:45,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:48,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:07:48,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:07:48,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:07:51,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:07:51,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:07:52,864 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:07:56,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:01,059 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:08:02,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:03,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:08:03,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:08:08,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:09,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:08:13,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:18,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:18,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:08:18,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:08:22,864 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:08:23,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:08:23,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:28,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:31,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:08:31,330 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:08:31,330 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:08:31,333 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:08:32,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:08:33,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:08:33,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:08:33,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:08:34,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:39,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:44,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:48,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:08:48,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:08:49,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:52,865 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:08:54,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:08:59,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:03,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:09:03,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:09:05,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:10,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:15,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:18,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:09:18,882 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:09:20,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:22,866 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:09:25,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:30,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:33,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:09:33,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:09:36,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:41,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:46,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:48,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:09:48,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:09:51,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:09:52,866 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:09:56,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:01,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:03,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:10:03,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:10:07,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:12,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:17,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:18,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:10:18,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:10:22,866 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:10:22,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:27,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:32,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:33,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:10:33,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:10:38,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:43,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:48,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:48,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:10:48,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:10:52,867 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:10:53,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:10:58,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:03,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:03,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:11:03,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:11:09,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:14,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:18,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:11:18,883 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:11:20,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:22,867 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:11:25,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:30,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:33,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:11:33,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:11:36,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:41,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:46,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:48,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:11:48,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:11:52,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:11:52,869 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:11:57,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:02,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:03,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:12:03,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:12:08,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:13,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:18,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:18,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:12:18,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:12:22,869 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:12:23,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:28,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:33,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:33,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:12:33,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:12:39,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:44,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:48,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:12:48,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:12:49,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:52,869 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:12:54,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:12:59,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:03,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:13:03,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:13:05,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:10,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:15,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:18,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:13:18,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:13:21,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:22,870 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:13:26,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:31,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:33,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:13:33,884 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:13:37,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:42,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:47,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:48,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:13:48,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:13:52,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:13:52,870 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:13:57,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:02,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:03,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:14:03,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:14:08,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:13,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:18,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:18,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:14:18,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:14:22,871 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:14:23,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:28,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:33,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:33,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:14:33,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:14:39,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:44,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:48,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:14:48,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:14:50,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:14:52,871 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:14:55,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:00,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:03,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:15:03,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:15:06,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:11,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:16,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:18,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:15:18,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:15:22,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:22,872 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:15:27,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:32,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:33,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:15:33,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:15:38,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:43,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:48,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:48,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:15:48,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:15:52,872 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:15:53,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:15:58,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:03,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:03,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:16:03,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:16:09,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:14,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:18,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:16:18,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:16:19,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:22,872 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:16:24,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:29,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:33,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:16:33,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:16:35,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:40,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:45,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:48,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:16:48,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:16:50,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:16:52,873 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:16:55,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:00,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:03,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:17:03,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:17:06,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:11,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:16,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:18,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:17:18,886 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:17:22,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:22,884 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:17:27,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:32,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:33,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:17:33,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:17:38,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:43,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:48,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:48,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:17:48,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:17:52,874 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:17:53,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:17:58,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:03,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:03,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:18:03,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:18:09,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:14,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:18,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:18:18,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:18:19,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:21,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:18:21,861 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:18:21,861 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:18:21,865 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:18:21,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:18:22,874 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:18:23,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:18:24,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:29,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:33,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:18:33,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:18:35,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:40,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:45,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:48,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:18:48,983 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:18:50,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:18:50,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:18:52,875 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:18:55,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:00,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:04,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:19:04,023 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:19:06,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:11,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:16,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:19,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:19:19,040 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:19:21,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:22,875 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:19:26,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:31,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:36,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:19:36,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:19:36,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:19:37,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:19:37,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:38,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:19:42,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:45,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:19:47,405 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:19:48,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:52,876 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:19:53,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:19:58,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:00,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:20:03,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:08,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:11,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:20:11,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:20:11,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:20:11,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:20:13,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:20:14,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:19,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:22,876 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:20:25,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:26,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:20:26,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:20:30,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:35,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:40,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:41,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:20:41,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:20:46,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:51,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:20:52,877 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:20:56,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:20:56,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:20:57,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:02,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:07,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:10,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:21:11,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:21:11,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:21:13,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:17,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:21:18,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:20,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:21:22,877 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:21:23,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:27,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:21:27,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:21:28,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:29,932 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:21:34,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:39,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:39,996 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:21:42,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:21:42,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:21:45,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:50,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:21:50,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:52,878 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:21:55,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:21:57,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:21:57,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:21:58,114 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:01,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:06,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:08,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:12,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:12,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:22:12,791 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:22:17,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:18,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:22,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:22,879 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:22:27,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:22:27,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:22:28,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:28,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:33,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:33,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:22:36,466 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:38,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:43,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:43,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:22:43,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:22:46,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:48,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:22:48,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:52,881 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:22:53,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:22:56,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:22:58,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:22:58,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:22:59,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:03,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:23:04,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:06,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:09,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:13,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:23:13,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:23:14,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:15,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:18,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:23:20,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:22,881 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:23:24,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:25,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:28,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:23:28,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:23:30,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:33,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:23:34,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:36,880 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:42,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:42,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:43,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:23:43,795 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:23:48,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:49,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:23:52,881 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:23:53,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:23:53,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:23:58,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:23:58,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:23:58,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:03,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:04,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:24:04,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:09,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:10,134 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:13,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:24:13,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:24:14,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:19,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:24:19,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:24:19,488 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:24:19,488 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:24:19,490 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:24:20,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:24:20,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:20,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:22,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:22,884 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:24:25,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:28,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:24:28,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:24:30,257 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:31,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:34,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:24:37,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:38,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:42,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:43,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:24:43,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:24:47,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:48,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:49,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:24:52,886 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:24:53,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:24:58,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:24:58,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:24:58,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:24:58,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:03,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:05,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:25:08,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:09,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:14,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:14,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:25:14,755 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:25:18,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:19,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:20,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:25:22,898 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:25:25,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:26,589 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:29,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:25:29,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:25:30,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:36,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:36,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:41,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:44,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:44,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:25:44,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:25:46,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:50,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:25:52,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:52,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:25:54,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:25:58,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:25:59,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:25:59,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:26:03,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:04,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:05,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:26:09,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:14,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:26:14,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:26:14,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:14,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:19,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:20,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:26:22,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:26:22,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:25,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:29,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:26:29,755 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:26:31,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:33,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:35,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:26:36,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:42,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:43,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:44,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:26:44,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:26:47,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:50,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:26:51,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:26:52,887 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:26:53,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:58,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:26:59,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:26:59,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:27:01,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:04,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:06,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:27:09,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:11,261 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:14,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:16,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:27:16,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:27:19,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:20,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:21,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:27:22,888 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:27:25,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:26,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:27:26,336 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:27:26,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:27:26,338 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:27:26,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:27:28,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:31,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:27:31,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:27:31,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:36,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:36,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:27:38,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:41,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:46,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:27:46,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:27:47,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:48,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:51,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:27:52,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:27:52,890 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:27:56,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:27:58,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:01,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:28:01,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:28:03,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:06,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:06,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:28:08,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:14,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:14,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:16,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:28:16,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:28:20,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:21,846 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:28:22,890 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:28:24,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:25,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:30,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:31,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:28:31,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:28:34,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:36,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:41,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:42,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:46,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:28:46,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:28:47,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:52,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:28:52,890 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:28:53,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:28:57,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:01,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:29:01,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:29:03,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:03,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:06,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:29:08,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:11,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:14,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:17,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:29:17,582 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:29:20,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:21,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:22,890 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:29:25,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:29,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:31,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:32,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:29:32,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:29:36,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:39,373 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:42,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:47,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:47,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:29:47,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:29:49,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:29:52,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:52,891 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:29:58,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:29:59,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:02,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:30:02,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:30:03,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:07,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:08,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:13,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:16,620 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:17,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:30:17,577 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:30:19,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:22,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:30:22,892 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:30:25,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:26,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:30,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:32,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:30:32,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:30:34,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:30:34,027 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:30:34,027 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:30:34,030 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:30:34,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:30:34,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:36,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:36,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:37,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:30:41,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:46,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:47,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:47,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:30:47,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:30:52,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:30:52,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:30:52,893 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:30:54,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:30:58,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:02,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:31:02,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:31:04,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:04,961 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:07,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:31:09,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:14,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:15,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:17,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:31:17,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:31:19,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:22,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:31:22,893 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:31:25,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:25,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:30,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:32,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:31:32,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:31:33,125 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:35,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:37,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:31:41,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:43,186 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:46,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:47,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:31:47,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:31:51,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:31:52,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:31:52,894 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:31:53,244 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:31:57,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:01,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:02,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:32:02,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:32:02,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:07,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:08,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:32:11,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:13,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:18,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:32:18,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:32:18,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:21,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:22,894 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:32:23,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:32:23,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:29,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:29,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:33,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:32:33,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:32:34,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:38,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:32:39,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:41,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:44,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:48,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:32:48,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:32:49,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:32:50,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:52,895 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:32:53,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:32:55,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:32:58,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:01,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:03,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:33:03,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:33:06,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:08,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:11,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:16,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:17,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:18,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:33:18,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:33:22,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:22,897 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:33:24,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:33:27,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:28,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:33,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:33,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:33:33,391 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:33:37,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:38,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:39,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:33:43,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:43,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:33:43,756 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:33:43,756 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:33:43,758 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:33:44,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:33:45,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:33:48,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:33:48,389 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:33:49,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:52,896 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:33:55,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:33:55,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:00,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:03,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:03,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:34:03,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:34:05,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:09,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:34:10,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:13,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:15,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:18,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:34:18,790 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:34:21,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:21,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:22,897 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:34:26,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:31,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:32,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:33,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:34:33,789 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:34:37,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:39,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:39,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:34:43,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:48,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:50,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:34:50,137 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:34:52,897 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:34:53,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:34:53,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:34:59,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:01,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:05,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:35:05,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:35:05,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:10,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:11,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:15,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:20,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:35:20,136 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:35:21,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:21,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:22,898 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:35:26,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:29,882 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:32,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:36,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:35:36,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:35:37,706 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:39,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:43,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:48,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:50,021 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:51,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:35:51,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:35:52,900 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:35:53,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:35:58,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:35:58,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:03,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:06,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:36:06,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:36:07,354 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:09,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:15,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:15,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:20,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:21,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:36:21,990 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:36:22,900 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:36:25,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:25,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:30,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:33,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:36,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:36,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:36:36,989 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:36:42,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:43,594 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:47,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:50,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:36:50,110 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:36:50,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:36:50,113 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:36:50,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:36:50,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:51,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:36:51,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:36:51,989 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:36:52,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:36:52,900 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:36:57,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:01,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:03,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:07,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:37:07,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:37:08,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:11,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:13,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:19,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:19,849 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:22,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:37:22,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:37:22,951 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:37:24,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:29,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:30,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:35,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:37,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:37:37,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:37:39,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:40,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:46,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:48,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:37:51,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:52,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:37:52,712 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:37:52,910 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:37:57,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:37:58,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:02,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:06,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:07,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:07,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:38:07,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:38:13,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:16,184 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:18,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:22,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:38:22,698 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:38:22,902 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:38:24,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:26,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:29,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:34,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:35,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:37,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:38:37,698 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:38:40,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:43,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:46,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:51,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:38:51,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:38:52,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:38:52,698 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:38:52,932 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:38:57,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:01,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:39:02,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:07,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:08,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:39:08,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:39:11,551 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:39:13,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:18,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:22,903 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:39:23,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:39:23,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:39:23,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:28,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:33,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:38,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:39:38,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:39:39,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:44,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:45,868 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:39:49,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:52,904 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:39:53,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:39:53,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:39:54,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:39:56,072 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:00,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:06,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:06,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:08,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:40:08,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:40:11,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:16,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:16,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:21,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:22,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:40:22,439 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:40:22,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:40:22,441 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:40:22,904 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:40:23,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:40:23,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:40:23,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:40:23,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:24,275 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:27,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:32,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:34,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:37,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:38,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:40:38,033 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:40:43,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:44,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:48,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:52,905 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:40:53,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:40:53,031 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:40:53,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:40:54,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:40:59,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:01,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:04,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:08,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:41:08,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:41:09,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:11,711 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:15,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:21,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:21,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:22,905 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:41:23,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:41:23,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:41:26,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:29,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:32,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:37,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:38,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:41:38,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:41:40,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:43,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:48,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:50,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:52,906 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:41:53,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:41:53,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:41:53,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:41:58,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:41:59,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:04,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:08,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:08,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:42:08,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:42:09,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:13,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:42:14,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:16,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:19,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:22,907 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:42:23,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:42:23,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:42:24,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:26,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:28,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:42:30,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:35,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:36,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:38,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:42:38,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:42:41,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:43,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:42:44,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:47,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:52,908 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:42:52,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:42:53,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:42:53,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:42:54,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:42:58,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:42:58,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:03,679 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:04,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:09,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:43:09,251 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:43:09,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:11,729 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:15,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:21,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:21,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:22,908 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:43:24,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:43:24,250 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:43:26,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:28,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:43:28,916 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:43:28,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:43:28,919 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:43:29,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:43:29,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:31,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:31,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:36,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:39,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:43:39,250 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:43:39,947 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:42,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:47,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:50,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:43:52,909 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:43:52,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:55,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:43:55,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:43:58,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:43:58,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:03,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:08,257 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:09,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:10,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:44:10,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:44:14,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:18,314 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:19,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:22,910 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:44:24,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:25,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:44:25,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:44:26,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:30,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:36,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:36,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:41,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:41,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:44:41,917 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:44:44,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:46,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:51,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:52,910 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:44:54,551 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:44:56,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:44:56,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:44:56,915 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:45:01,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:02,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:06,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:11,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:45:11,916 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:45:12,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:12,667 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:17,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:22,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:22,911 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:45:22,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:27,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:28,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:45:28,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:45:32,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:33,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:39,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:40,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:43,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:45:43,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:45:44,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:49,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:49,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:52,911 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:45:55,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:45:57,995 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:45:58,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:45:58,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:46:01,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:06,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:08,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:12,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:13,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:46:13,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:46:17,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:18,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:22,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:22,913 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:46:26,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:28,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:28,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:46:28,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:46:34,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:34,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:46:34,296 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:46:34,297 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:46:34,298 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:46:35,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:46:35,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:36,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:43,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:46:43,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:46:45,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:46,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:50,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:52,913 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:46:54,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:46:55,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:46:58,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:46:58,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:47:00,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:04,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:06,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:11,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:12,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:13,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:47:13,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:47:17,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:22,681 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:22,913 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:47:22,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:27,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:28,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:47:28,327 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:47:32,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:33,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:38,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:42,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:43,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:43,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:47:43,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:47:49,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:50,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:47:52,914 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:47:54,915 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:47:58,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:47:58,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:48:00,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:00,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:06,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:08,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:11,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:13,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:48:13,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:48:16,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:19,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:21,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:22,916 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:48:26,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:28,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:48:28,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:48:29,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:32,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:36,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:37,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:42,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:43,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:48:43,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:48:48,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:50,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:48:52,916 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:48:53,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:48:58,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:48:58,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:48:59,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:00,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:05,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:10,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:11,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:13,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:49:13,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:49:16,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:18,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:22,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:22,916 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:49:27,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:28,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:49:28,326 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:49:28,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:32,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:37,978 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:38,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:42,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:44,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:49:44,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:49:45,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:49:45,720 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:49:45,721 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:49:45,723 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:49:46,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:49:46,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:47,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:48,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:52,918 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:49:53,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:49:56,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:49:59,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:49:59,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:49:59,826 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:05,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:06,755 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:10,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:14,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:50:14,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:50:14,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:15,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:20,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:21,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:21,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:22,918 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:50:26,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:29,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:50:29,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:50:30,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:32,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:37,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:38,979 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:42,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:45,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:50:45,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:50:48,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:49,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:52,919 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:50:53,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:50:57,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:50:59,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:00,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:51:00,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:51:05,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:07,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:10,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:15,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:15,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:51:15,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:51:16,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:21,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:22,919 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:51:25,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:26,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:31,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:51:31,277 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:51:31,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:51:32,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:35,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:37,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:42,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:42,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:46,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:51:46,276 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:51:48,495 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:52,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:51:52,920 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:51:53,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:51:58,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:00,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:01,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:52:01,277 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:52:04,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:09,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:10,556 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:14,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:16,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:52:16,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:52:16,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:52:20,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:20,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:22,921 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:52:25,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:28,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:31,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:31,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:52:31,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:52:36,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:38,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:41,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:46,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:52:46,668 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:52:46,669 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:52:46,671 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:52:46,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:52:46,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:52:46,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:52:47,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:47,801 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:48,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:52:52,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:52,923 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:52:57,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:52:58,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:01,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:53:01,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:53:03,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:06,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:08,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:13,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:16,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:53:16,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:53:16,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:19,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:22,923 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:53:25,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:25,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:30,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:32,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:53:32,059 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:53:35,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:35,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:40,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:45,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:45,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:47,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:53:47,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:53:51,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:53:52,923 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:53:53,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:53:56,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:02,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:54:02,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:02,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:54:03,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:07,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:11,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:12,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:17,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:18,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:54:18,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:54:18,040 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:54:20,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:22,924 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:54:22,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:27,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:30,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:33,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:54:33,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:33,038 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:54:38,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:38,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:43,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:48,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:54:48,038 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:54:48,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:54:49,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:52,924 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:54:54,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:54:58,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:00,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:04,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:55:04,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:55:04,561 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:55:05,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:06,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:10,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:16,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:16,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:19,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:55:19,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:55:19,560 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:55:22,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:22,925 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:55:25,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:28,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:33,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:34,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:55:34,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:55:34,560 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:55:35,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:39,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:43,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:44,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:49,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:55:49,893 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:55:49,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:55:50,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:55:51,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:55:51,006 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:55:51,007 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:55:51,009 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:55:51,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:55:52,248 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:52,925 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:55:53,252 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:55:55,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:01,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:01,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:04,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:56:04,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:56:07,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:11,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:12,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:17,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:19,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:56:19,892 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:56:21,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:22,926 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:56:22,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:27,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:31,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:33,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:36,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:56:36,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:56:36,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:56:38,737 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:39,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:44,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:48,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:49,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:51,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:56:51,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:56:52,927 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:56:54,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:56:56,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:56:59,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:04,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:06,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:57:06,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:57:06,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:57:10,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:14,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:57:16,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:21,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:22,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:57:22,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:57:22,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:57:22,927 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:57:25,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:57:27,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:32,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:37,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:57:37,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:57:37,075 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:57:37,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:42,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:45,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:57:47,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:52,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:57:52,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:57:52,078 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:57:52,928 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:57:53,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:57:55,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:57:58,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:04,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:05,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:07,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:58:07,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:58:07,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:58:10,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:15,428 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:15,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:20,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:22,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:58:22,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:58:22,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:58:22,929 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:58:23,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:26,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:32,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:33,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:37,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:58:37,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:58:37,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:58:37,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:41,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:43,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:48,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:51,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:58:52,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:58:52,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:58:52,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:58:52,929 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:58:53,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:58:59,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:01,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:04,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:07,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:59:07,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:59:07,078 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:59:07,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 04:59:07,832 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 04:59:07,833 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 04:59:07,836 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 04:59:08,796 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 04:59:09,800 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:09,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:10,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:14,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:19,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:20,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:22,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:59:22,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:59:22,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:59:22,930 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:59:25,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:27,909 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:31,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:37,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:59:37,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:59:37,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:59:37,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:37,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:42,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:47,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:48,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:52,931 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 04:59:53,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 04:59:53,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 04:59:53,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 04:59:53,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 04:59:55,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 04:59:59,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:04,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:05,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:08,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:00:08,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:00:10,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:13,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:15,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:20,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:22,931 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:00:23,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:00:23,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:00:23,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:26,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:31,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:33,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:37,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:38,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:00:38,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:00:38,097 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:00:42,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:43,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:47,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:51,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:00:52,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:00:52,932 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:00:53,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:00:53,094 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:00:58,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:01,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:03,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:08,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:01:08,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:01:08,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:01:09,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:11,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:14,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:19,662 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:19,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:22,932 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:01:23,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:01:23,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:01:23,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:01:25,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:29,722 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:30,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:35,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:37,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:38,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:01:38,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:01:38,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:01:41,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:46,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:47,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:01:51,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:52,933 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:01:53,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:01:53,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:01:53,869 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:01:56,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:01:57,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:01,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:06,003 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:06,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:08,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:02:08,860 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:02:12,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:13,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:02:13,056 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:02:13,057 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:02:13,059 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:02:14,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:02:15,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:16,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:18,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:22,934 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:02:23,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:23,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:02:23,860 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:02:24,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:29,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:34,193 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:34,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:38,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:02:38,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:02:40,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:45,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:47,270 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:50,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:02:52,935 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:02:53,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:02:53,869 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:02:55,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:02:55,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:00,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:05,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:06,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:08,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:03:08,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:03:12,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:15,448 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:17,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:22,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:22,935 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:03:23,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:03:23,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:03:25,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:28,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:33,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:33,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:38,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:38,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:03:38,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:03:43,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:44,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:49,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:03:52,936 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:03:53,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:03:53,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:03:53,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:03:55,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:00,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:03,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:05,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:09,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:04:09,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:04:09,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:04:11,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:11,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:16,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:21,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:21,891 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:22,937 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:04:24,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:04:24,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:04:24,160 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:04:27,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:29,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:33,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:38,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:39,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:04:39,158 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:04:40,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:43,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:48,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:48,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:52,938 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:04:53,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:04:54,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:04:54,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:04:54,557 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:04:58,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:04:59,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:04,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:08,209 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:09,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:05:09,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:05:09,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:05:10,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:16,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:16,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:21,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:22,938 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:05:23,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:05:23,349 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:05:23,350 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:05:23,352 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:05:24,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:05:24,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:05:24,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:05:24,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:05:25,329 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:26,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:26,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:31,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:34,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:36,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:40,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:05:40,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:05:40,755 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:05:42,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:43,475 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:47,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:51,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:05:52,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:05:52,939 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:05:55,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:05:55,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:05:55,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:05:57,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:01,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:03,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:08,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:09,628 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:10,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:06:10,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:06:10,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:06:14,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:19,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:19,686 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:22,939 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:06:24,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:26,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:06:26,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:06:26,264 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:06:29,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:30,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:35,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:37,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:40,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:41,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:06:41,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:06:41,262 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:06:45,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:47,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:50,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:06:52,941 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:06:55,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:06:56,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:06:56,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:06:56,262 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:06:56,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:01,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:06,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:07,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:12,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:07:12,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:07:12,187 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:07:12,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:16,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:18,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:22,940 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:07:23,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:24,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:27,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:07:27,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:07:27,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:07:29,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:34,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:34,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:39,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:42,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:07:42,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:07:42,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:07:42,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:45,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:50,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:52,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:07:52,941 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:07:55,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:07:57,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:07:57,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:07:57,453 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:08:00,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:01,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:06,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:10,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:11,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:12,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:08:12,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:08:12,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:08:16,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:20,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:22,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:22,942 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:08:27,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:08:27,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:08:27,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:27,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:08:32,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:37,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:40,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:08:40,597 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:08:40,597 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:08:40,602 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:08:40,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:08:41,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:42,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:08:42,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:08:42,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:08:42,635 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:42,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:47,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:51,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:08:52,943 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:08:52,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:08:57,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:08:57,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:08:57,454 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:08:58,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:01,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:04,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:09,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:11,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:12,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:09:12,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:09:12,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:09:14,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:19,911 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:20,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:22,943 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:09:25,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:27,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:09:27,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:09:27,487 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:09:29,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:31,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:36,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:40,042 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:41,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:42,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:09:42,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:09:47,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:48,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:52,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:09:52,944 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:09:57,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:09:57,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:09:58,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:09:58,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:03,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:06,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:09,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:13,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:10:13,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:10:13,485 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:10:14,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:16,273 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:19,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:22,946 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:10:24,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:26,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:28,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:10:28,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:10:30,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:34,389 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:35,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:40,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:43,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:10:43,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:10:44,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:46,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:52,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:52,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:10:52,946 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:10:57,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:10:59,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:10:59,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:10:59,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:11:02,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:03,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:08,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:10,636 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:13,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:14,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:11:14,395 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:11:19,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:20,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:22,946 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:11:24,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:28,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:29,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:11:29,395 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:11:30,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:35,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:38,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:41,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:45,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:11:45,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:11:45,321 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:11:46,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:46,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:11:46,582 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:11:46,582 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:11:46,584 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:11:46,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:11:46,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:48,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:51,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:11:52,947 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:11:55,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:11:57,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:00,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:12:00,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:12:02,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:07,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:12,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:15,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:12:15,319 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:12:18,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:22,948 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:12:23,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:28,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:30,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:12:30,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:12:30,320 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:12:34,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:38,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:12:40,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:45,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:45,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:12:45,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:12:45,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:12:48,377 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:12:50,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:52,948 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:12:55,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:12:58,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:00,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:13:00,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:13:00,437 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:13:01,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:07,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:08,524 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:12,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:15,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:13:15,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:13:15,438 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:13:17,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:18,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:22,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:22,950 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:13:26,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:28,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:30,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:13:30,438 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:13:33,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:36,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:39,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:45,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:45,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:13:45,438 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:13:46,765 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:50,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:13:52,950 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:13:54,814 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:13:56,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:01,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:14:01,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:14:01,223 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:14:01,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:04,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:07,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:12,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:12,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:16,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:14:16,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:14:17,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:22,950 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:14:22,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:22,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:27,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:31,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:31,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:14:31,221 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:14:33,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:38,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:41,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:44,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:47,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:14:47,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:14:49,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:50,163 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:14:52,951 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:14:54,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:14:58,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:00,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:02,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:15:02,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:15:06,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:08,275 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:11,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:16,321 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:16,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:17,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:15:17,086 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:15:22,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:22,952 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:15:25,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:15:25,655 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:15:25,656 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:15:25,657 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:15:26,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:15:26,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:27,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:27,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:32,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:15:32,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:15:32,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:15:33,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:36,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:39,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:44,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:44,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:47,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:15:47,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:15:50,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:15:52,953 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:15:54,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:15:55,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:00,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:02,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:16:02,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:16:02,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:06,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:11,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:12,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:16,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:17,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:16:17,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:16:21,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:22,759 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:22,953 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:16:26,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:32,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:32,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:16:32,087 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:16:32,829 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:37,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:40,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:42,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:47,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:16:47,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:16:47,838 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:16:48,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:50,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:16:52,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:16:53,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:16:59,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:01,284 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:02,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:17:02,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:17:02,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:17:05,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:08,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:11,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:16,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:17,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:17:17,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:17:21,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:22,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:22,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:17:26,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:32,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:32,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:32,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:17:32,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:17:38,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:42,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:43,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:48,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:17:48,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:17:48,156 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:17:48,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:50,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:17:52,954 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:17:53,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:17:59,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:00,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:03,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:18:03,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:18:03,155 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:18:05,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:08,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:11,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:16,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:18,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:18:18,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:18:18,154 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:18:18,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:21,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:22,955 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:18:26,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:27,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:32,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:34,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:18:34,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:18:34,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:18:35,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:18:35,838 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:18:35,839 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:18:35,841 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:18:35,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:18:36,968 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:37,842 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:37,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:42,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:47,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:47,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:49,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:18:49,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:18:49,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:18:52,957 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:18:53,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:18:55,089 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:18:59,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:04,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:04,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:19:04,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:19:04,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:19:05,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:09,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:15,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:15,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:19,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:19:19,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:19:19,735 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:19:20,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:22,957 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:19:25,378 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:25,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:30,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:33,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:34,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:19:34,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:19:34,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:19:35,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:41,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:43,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:46,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:49,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:19:49,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:19:49,734 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:19:50,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:19:51,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:19:52,957 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:19:56,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:00,612 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:02,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:04,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:20:04,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:20:04,734 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:20:07,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:10,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:13,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:18,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:19,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:19,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:20:19,734 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:20:22,958 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:20:24,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:28,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:30,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:34,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:20:34,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:20:34,975 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:20:35,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:36,862 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:41,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:46,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:47,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:49,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:20:49,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:20:52,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:20:52,959 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:20:56,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:20:58,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:03,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:04,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:21:04,973 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:21:05,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:08,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:14,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:15,120 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:19,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:20,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:21:20,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:21:20,760 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:21:22,959 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:21:23,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:24,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:29,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:33,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:35,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:35,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:21:35,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:21:40,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:21:40,196 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:21:40,197 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:21:40,198 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:21:40,292 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:21:41,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:41,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:42,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:46,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:50,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:21:50,759 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:21:51,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:21:52,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:52,959 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:21:57,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:21:59,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:03,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:06,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:22:06,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:22:06,128 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:22:08,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:09,494 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:13,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:18,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:19,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:21,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:22:21,127 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:22:22,960 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:22:23,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:27,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:29,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:34,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:36,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:22:36,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:22:36,128 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:22:37,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:40,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:45,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:45,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:50,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:22:52,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:22:52,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:22:52,219 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:22:52,960 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:22:54,787 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:22:56,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:01,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:04,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:06,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:07,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:23:07,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:23:07,218 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:23:12,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:12,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:17,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:22,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:23:22,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:23:22,217 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:23:22,961 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:23:22,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:22,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:27,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:31,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:33,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:38,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:23:38,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:23:38,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:23:38,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:41,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:43,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:48,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:51,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:23:52,964 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:23:53,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:23:53,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:23:53,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:23:53,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:23:58,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:01,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:03,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:08,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:24:08,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:24:08,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:24:09,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:09,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:14,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:19,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:20,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:22,964 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:24:23,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:24:23,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:24:23,178 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:24:26,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:27,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:31,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:36,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:38,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:24:38,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:24:38,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:24:39,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:42,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:46,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:24:46,257 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:24:46,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:24:46,259 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:24:46,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:24:47,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:48,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:48,598 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:52,966 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:24:53,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:24:54,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:24:54,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:24:54,055 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:24:57,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:24:59,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:04,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:05,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:09,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:25:09,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:25:09,053 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:25:09,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:14,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:15,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:19,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:22,966 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:25:24,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:25:24,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:25:24,053 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:25:24,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:25,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:29,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:33,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:35,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:40,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:40,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:25:40,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:25:40,307 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:25:43,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:45,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:50,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:25:50,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:25:52,967 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:25:55,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:25:55,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:25:55,302 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:25:56,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:01,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:01,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:06,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:10,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:26:10,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:26:10,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:26:11,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:12,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:17,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:19,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:22,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:22,976 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:26:26,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:26:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:26:26,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:26:27,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:29,250 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:33,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:38,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:39,318 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:41,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:26:41,456 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:26:43,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:47,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:26:49,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:52,978 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:26:54,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:26:56,455 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:26:56,455 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:26:57,440 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:27:00,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:05,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:10,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:11,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:27:11,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:27:15,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:20,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:22,978 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:27:25,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:26,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:27:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:27:26,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:27:29,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:27:31,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:36,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:39,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:27:41,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:27:41,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:27:42,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:47,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:27:48,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:52,980 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:27:53,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:27:56,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:27:56,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:27:57,825 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:27:59,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:04,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:07,905 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:10,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:11,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:28:11,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:28:14,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:28:14,521 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:28:14,522 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:28:14,524 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:28:14,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:28:15,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:15,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:16,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:20,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:22,981 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:28:26,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:26,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:26,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:28:26,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:28:31,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:34,081 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:37,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:41,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:28:41,459 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:28:41,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:28:42,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:44,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:47,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:52,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:28:52,982 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:28:53,236 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:28:56,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:28:56,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:28:57,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:03,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:03,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:08,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:11,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:29:11,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:29:13,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:13,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:18,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:21,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:22,982 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:29:23,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:26,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:29:26,461 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:29:29,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:31,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:34,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:39,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:39,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:41,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:29:41,459 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:29:44,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:49,618 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:29:50,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:52,982 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:29:55,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:29:56,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:29:56,460 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:29:57,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:01,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:06,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:07,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:11,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:30:11,459 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:30:11,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:16,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:17,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:21,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:22,983 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:30:26,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:30:26,459 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:30:27,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:27,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:32,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:35,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:38,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:42,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:30:42,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:30:42,423 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:30:43,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:46,002 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:48,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:52,984 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:30:54,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:30:54,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:30:57,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:30:57,419 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:30:59,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:04,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:04,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:09,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:12,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:31:12,419 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:31:15,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:18,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:20,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:22,986 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:31:25,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:26,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:31:26,015 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:31:26,016 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:31:26,018 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:31:26,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:31:27,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:31:27,419 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:31:28,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:31,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:36,340 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:37,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:42,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:43,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:31:43,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:31:43,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:31:46,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:48,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:52,987 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:31:53,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:31:55,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:31:58,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:31:58,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:31:58,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:31:59,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:03,512 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:04,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:09,741 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:13,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:32:13,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:32:13,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:32:13,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:15,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:20,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:22,987 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:32:23,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:25,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:29,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:32:29,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:32:29,228 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:32:31,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:31,682 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:36,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:41,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:41,928 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:44,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:32:44,226 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:32:47,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:49,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:32:52,987 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:32:52,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:58,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:32:59,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:32:59,226 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:32:59,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:04,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:07,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:09,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:14,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:33:14,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:33:14,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:33:15,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:17,994 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:20,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:22,989 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:33:26,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:28,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:29,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:33:29,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:33:29,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:33:31,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:36,107 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:37,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:42,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:44,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:33:44,801 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:33:46,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:48,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:52,989 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:33:53,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:33:54,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:33:58,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:01,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:34:01,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:34:01,048 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:34:04,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:04,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:09,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:14,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:15,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:16,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:34:16,000 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:34:20,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:22,990 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:34:24,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:25,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:30,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:34:30,382 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:34:30,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:34:30,385 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:34:30,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:34:30,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:34:31,000 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:34:31,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:32,469 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:36,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:41,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:42,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:46,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:47,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:34:47,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:34:47,192 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:34:50,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:34:52,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:34:52,990 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:34:57,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:00,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:02,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:35:02,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:35:02,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:35:02,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:07,721 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:08,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:13,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:17,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:35:17,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:35:17,190 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:35:17,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:18,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:22,991 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:35:23,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:25,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:29,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:32,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:35:32,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:35:32,194 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:35:33,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:35,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:40,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:43,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:46,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:47,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:35:47,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:35:51,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:35:53,002 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:35:54,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:35:57,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:02,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:02,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:02,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:36:02,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:36:07,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:12,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:12,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:17,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:36:17,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:36:17,501 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:36:17,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:20,173 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:22,992 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:36:22,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:27,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:30,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:32,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:36:32,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:36:32,499 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:36:33,676 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:38,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:39,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:44,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:47,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:36:47,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:36:47,499 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:36:48,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:36:49,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:52,994 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:36:54,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:36:56,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:00,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:02,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:37:02,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:37:02,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:37:05,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:06,482 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:10,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:14,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:15,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:17,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:37:17,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:37:17,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:37:21,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:22,994 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:37:24,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:26,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:30,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:37:30,687 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:37:30,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:37:30,689 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:37:31,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:37:32,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:32,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:32,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:37:32,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:37:32,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:37:33,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:38,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:42,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:43,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:47,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:37:47,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:37:47,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:37:48,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:50,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:37:52,994 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:37:53,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:37:59,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:00,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:02,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:38:02,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:38:02,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:38:05,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:10,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:10,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:15,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:17,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:38:17,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:38:17,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:38:20,940 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:21,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:22,995 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:38:26,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:27,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:32,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:32,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:38:32,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:38:32,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:38:38,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:38,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:43,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:47,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:38:47,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:38:47,864 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:38:48,105 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:49,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:52,997 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:38:54,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:38:56,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:38:59,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:02,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:39:02,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:39:02,868 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:39:04,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:05,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:10,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:14,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:16,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:17,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:39:17,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:39:17,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:39:22,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:22,997 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:39:24,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:28,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:32,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:32,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:39:32,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:39:32,865 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:39:33,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:38,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:42,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:43,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:48,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:39:48,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:39:48,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:39:48,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:50,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:39:52,997 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:39:53,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:39:59,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:00,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:03,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:40:03,471 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:40:04,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:09,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:10,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:14,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:18,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:40:18,470 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:40:18,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:20,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:22,998 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:40:25,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:28,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:31,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:34,509 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:40:34,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:40:34,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:40:35,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:40:35,706 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:40:35,707 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:40:35,709 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:40:35,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:40:36,710 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:36,840 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:37,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:41,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:46,902 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:47,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:49,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:40:49,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:40:49,511 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:40:52,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:40:53,000 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:40:56,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:40:58,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:03,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:04,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:41:04,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:41:04,511 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:41:07,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:08,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:14,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:15,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:19,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:41:19,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:41:19,514 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:41:19,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:22,999 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:41:25,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:25,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:30,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:33,197 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:34,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:41:34,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:41:34,511 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:41:35,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:41,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:42,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:46,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:49,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:41:49,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:41:49,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:41:50,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:41:51,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:41:53,000 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:41:57,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:00,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:02,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:04,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:42:04,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:42:04,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:42:07,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:10,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:13,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:18,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:18,515 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:19,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:42:19,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:42:19,511 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:42:23,000 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:42:24,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:28,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:29,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:34,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:35,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:42:35,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:42:35,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:42:38,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:39,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:44,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:48,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:42:49,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:50,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:42:50,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:42:50,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:42:53,003 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:42:55,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:42:56,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:00,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:05,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:05,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:43:05,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:43:05,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:43:06,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:10,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:14,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:16,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:21,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:21,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:43:21,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:43:21,879 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:43:23,002 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:43:24,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:27,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:32,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:32,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:36,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:43:36,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:43:38,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:41,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:43:41,168 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:43:41,169 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:43:41,171 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:43:42,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:43:43,054 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:43,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:44,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:48,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:51,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:43:51,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:43:53,003 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:43:53,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:43:54,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:43:59,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:01,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:04,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:07,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:44:07,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:44:07,069 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:44:10,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:11,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:15,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:19,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:20,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:22,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:44:22,067 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:44:23,004 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:44:26,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:29,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:31,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:36,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:37,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:44:37,068 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:44:37,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:42,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:47,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:50,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:44:52,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:44:52,068 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:44:52,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:53,004 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:44:58,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:44:58,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:03,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:07,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:45:07,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:45:07,069 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:45:08,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:08,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:13,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:18,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:19,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:22,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:45:22,068 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:45:23,007 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:45:25,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:26,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:30,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:36,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:36,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:37,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:45:37,069 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:45:41,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:44,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:46,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:51,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:45:52,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:45:52,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:45:52,091 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:45:53,007 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:45:54,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:45:57,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:02,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:04,989 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:07,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:46:07,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:46:07,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:46:07,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:12,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:13,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:17,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:22,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:46:22,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:46:23,007 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:46:23,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:23,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:28,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:31,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:33,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:37,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:46:37,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:46:37,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:46:38,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:41,262 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:43,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:48,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:46:48,328 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:46:48,328 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:46:48,331 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:46:49,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:46:49,318 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:49,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:51,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:46:52,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:46:52,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:46:53,210 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:46:55,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:46:59,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:00,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:05,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:07,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:47:07,991 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:47:09,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:11,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:16,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:17,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:21,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:23,010 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:47:23,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:47:23,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:47:23,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:47:26,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:27,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:31,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:35,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:37,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:38,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:47:38,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:47:38,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:47:43,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:45,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:48,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:52,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:47:53,012 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:47:53,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:47:53,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:47:53,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:47:53,924 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:47:59,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:02,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:05,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:10,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:48:10,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:48:10,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:48:10,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:12,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:15,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:20,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:21,611 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:23,010 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:48:25,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:48:25,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:48:25,034 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:48:27,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:31,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:32,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:38,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:39,069 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:40,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:48:40,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:48:40,034 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:48:44,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:47,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:48:50,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:53,013 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:48:55,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:48:55,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:48:55,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:48:55,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:48:57,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:00,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:06,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:07,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:10,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:49:10,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:49:10,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:49:12,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:15,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:17,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:23,012 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:49:23,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:23,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:25,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:49:25,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:49:25,034 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:49:28,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:33,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:34,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:39,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:40,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:49:40,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:49:40,037 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:49:43,550 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:44,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:49,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:49:49,628 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:49:49,628 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:49:49,630 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:49:50,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:49:50,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:49:51,609 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:49:53,013 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:49:55,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:49:55,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:49:55,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:49:56,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:01,635 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:01,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:06,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:09,801 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:10,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:50:10,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:50:10,035 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:50:12,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:17,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:19,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:22,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:23,014 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:50:25,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:50:25,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:50:25,038 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:50:27,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:27,928 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:32,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:38,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:38,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:40,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:50:40,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:50:40,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:50:43,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:45,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:50:49,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:53,015 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:50:54,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:50:55,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:50:55,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:50:55,036 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:50:55,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:00,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:03,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:05,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:10,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:51:10,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:51:10,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:51:11,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:13,326 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:16,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:21,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:21,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:23,015 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:51:25,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:51:25,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:51:25,075 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:51:27,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:31,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:32,353 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:37,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:39,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:40,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:51:40,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:51:40,075 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:51:43,328 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:48,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:49,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:53,018 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:51:54,021 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:51:55,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:51:55,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:51:55,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:51:57,665 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:51:59,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:04,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:07,735 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:10,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:52:10,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:10,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:52:10,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:52:15,791 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:15,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:20,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:23,017 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:52:23,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:25,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:52:25,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:52:25,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:52:26,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:31,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:33,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:36,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:40,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:52:40,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:52:40,079 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:52:42,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:43,994 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:47,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:49,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:52:49,836 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:52:49,837 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:52:49,839 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:52:50,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:52:52,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:52:52,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:52:53,018 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:52:55,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:52:55,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:52:55,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:52:58,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:02,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:04,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:10,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:10,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:53:10,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:53:10,077 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:53:10,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:15,266 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:19,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:21,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:23,018 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:53:25,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:53:25,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:53:25,186 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:53:26,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:27,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:31,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:37,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:37,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:40,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:53:40,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:53:42,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:47,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:48,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:53,021 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:53:54,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:53:55,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:53:55,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:53:55,185 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:53:55,487 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:53:59,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:05,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:05,548 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:10,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:10,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:54:10,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:54:10,950 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:54:13,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:15,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:21,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:21,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:23,021 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:54:25,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:54:25,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:54:25,948 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:54:27,098 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:31,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:32,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:37,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:40,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:54:40,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:54:40,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:54:41,788 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:42,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:47,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:49,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:52,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:54:53,021 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:54:55,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:54:55,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:54:55,950 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:54:57,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:54:58,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:03,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:07,990 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:08,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:10,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:55:10,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:55:14,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:16,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:19,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:23,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:55:25,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:25,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:55:25,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:55:26,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:30,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:34,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:35,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:40,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:40,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:55:40,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:55:40,951 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:55:44,234 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:45,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:50,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:55:50,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:55:50,942 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:55:50,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:55:50,945 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:55:51,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:55:52,288 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:55:53,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:55:55,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:55:55,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:55:56,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:02,016 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:02,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:07,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:10,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:10,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:56:10,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:56:12,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:17,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:19,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:22,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:23,023 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:56:25,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:56:25,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:56:25,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:56:27,523 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:28,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:33,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:37,581 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:39,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:40,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:56:40,950 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:56:45,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:45,632 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:50,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:56:53,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:56:55,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:56:55,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:56:55,950 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:56:56,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:01,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:03,743 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:07,096 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:11,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:57:11,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:57:11,162 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:57:12,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:13,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:17,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:22,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:23,025 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:57:23,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:26,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:57:26,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:57:26,160 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:57:27,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:31,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:32,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:37,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:41,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:57:41,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:57:41,160 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:57:42,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:43,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:48,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:50,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:57:53,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:57:54,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:57:57,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:57:57,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:57:57,416 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:57:59,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:00,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:04,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:08,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:10,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:12,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:58:12,414 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:58:15,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:20,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:22,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:23,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:58:26,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:28,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:58:28,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:58:28,635 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:58:31,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:32,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:36,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:40,389 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:42,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:43,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:58:43,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:58:43,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:58:47,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:49,711 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:53,026 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:58:53,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:57,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 05:58:57,345 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 05:58:57,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 05:58:57,348 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 05:58:57,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 05:58:58,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:58:58,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:58:58,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:58:58,633 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:58:58,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:58:59,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:03,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:07,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:09,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:13,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:59:13,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:59:13,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:59:15,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:17,884 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:20,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:23,028 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:59:25,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:26,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:28,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:59:28,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:59:28,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:59:32,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:35,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:37,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:43,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:43,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:59:43,929 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 05:59:44,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:49,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:53,028 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 05:59:54,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 05:59:55,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 05:59:59,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 05:59:59,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 05:59:59,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:00:00,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:02,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:00:05,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:10,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:12,225 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:00:14,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:00:14,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:00:16,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:20,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:00:22,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:23,029 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:00:28,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:29,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:00:29,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:00:30,367 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:00:33,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:38,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:40,459 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:00:43,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:00:44,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:00:49,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:53,029 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:00:55,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:00:59,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:00:59,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:01:00,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:06,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:06,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:11,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:14,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:01:14,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:01:14,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:01:16,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:16,677 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:21,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:23,030 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:01:24,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:27,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:29,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:01:29,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:01:32,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:34,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:37,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:42,847 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:43,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:01:44,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:01:48,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:51,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:01:53,030 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:01:54,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:59,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:01:59,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:01:59,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:02:01,963 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:04,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:10,011 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:10,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:14,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:02:14,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:02:15,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:17,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:02:17,827 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:02:17,827 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:02:17,828 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:02:18,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:02:19,075 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:20,080 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:20,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:23,031 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:02:26,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:28,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:29,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:02:29,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:02:29,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:02:31,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:37,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:38,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:42,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:44,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:02:44,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:02:46,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:47,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:52,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:53,033 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:02:56,464 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:02:58,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:02:59,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:02:59,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:03:04,429 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:04,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:09,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:12,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:14,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:03:14,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:03:14,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:19,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:22,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:23,040 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:03:25,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:29,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:03:29,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:03:30,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:30,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:35,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:40,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:40,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:44,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:03:44,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:03:46,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:50,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:52,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:53,041 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:03:58,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:03:59,022 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:03:59,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:03:59,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:04:03,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:08,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:09,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:13,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:14,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:04:14,389 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:04:16,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:19,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:23,042 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:04:25,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:26,215 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:29,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:04:29,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:04:30,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:36,285 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:36,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:41,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:44,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:44,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:04:44,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:04:46,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:51,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:04:53,042 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:04:54,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:04:57,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:00,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:05:00,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:05:00,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:05:02,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:02,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:07,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:12,520 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:12,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:15,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:05:15,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:05:15,091 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:05:18,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:19,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:05:19,476 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:05:19,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:05:19,478 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:05:19,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:05:20,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:23,043 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:05:24,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:29,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:30,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:05:30,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:05:30,091 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:05:30,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:35,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:38,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:40,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:45,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:46,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:05:46,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:05:46,197 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:05:48,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:05:51,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:53,044 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:05:56,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:05:58,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:01,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:06:01,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:06:02,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:06,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:07,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:12,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:16,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:06:16,195 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:06:16,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:18,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:23,045 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:06:23,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:24,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:28,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:31,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:06:31,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:06:31,922 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:06:33,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:35,050 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:38,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:44,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:45,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:46,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:06:46,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:06:46,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:06:50,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:06:53,045 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:06:53,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:06:56,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:01,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:01,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:07:01,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:07:01,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:07:03,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:07,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:11,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:12,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:17,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:17,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:07:17,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:07:17,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:07:21,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:22,736 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:23,056 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:07:28,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:28,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:32,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:07:32,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:07:32,685 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:07:33,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:38,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:39,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:44,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:47,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:07:47,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:07:47,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:07:50,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:07:50,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:07:53,046 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:07:56,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:01,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:02,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:08:02,685 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:08:06,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:11,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:14,734 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:08:17,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:17,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:08:17,685 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:08:22,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:22,792 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:08:23,047 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:08:28,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:32,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:08:32,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:08:32,863 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:08:33,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:38,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:39,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:08:39,376 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:08:39,376 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:08:39,378 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:08:39,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:08:40,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:08:44,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:47,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:08:47,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:08:47,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:08:49,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:50,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:08:53,047 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:08:55,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:08:59,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:00,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:02,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:09:02,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:09:06,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:09,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:11,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:17,171 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:17,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:17,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:09:17,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:09:22,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:23,048 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:09:27,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:28,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:33,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:09:33,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:09:33,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:09:33,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:37,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:39,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:45,352 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:45,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:48,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:09:48,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:09:48,553 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:09:50,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:09:53,048 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:09:54,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:09:56,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:01,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:02,683 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:03,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:10:03,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:10:03,554 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:10:06,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:10,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:12,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:17,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:18,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:10:18,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:10:18,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:10:20,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:22,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:23,049 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:10:28,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:30,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:33,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:33,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:10:33,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:10:33,746 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:10:39,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:40,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:44,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:48,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:10:48,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:10:48,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:10:48,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:10:49,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:53,049 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:10:55,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:10:59,057 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:00,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:05,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:11:05,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:11:05,155 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:11:06,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:07,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:11,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:17,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:17,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:20,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:11:20,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:11:22,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:23,051 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:11:27,237 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:27,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:32,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:35,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:11:35,153 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:11:35,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:38,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:42,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:11:42,693 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:11:42,694 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:11:42,698 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:11:43,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:11:43,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:43,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:45,349 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:48,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:50,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:11:50,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:11:50,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:11:53,051 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:11:53,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:11:54,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:11:59,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:03,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:05,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:05,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:12:05,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:12:05,885 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:12:10,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:11,627 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:15,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:20,692 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:20,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:12:20,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:12:20,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:12:23,052 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:12:25,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:26,544 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:31,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:34,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:35,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:12:35,887 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:12:37,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:42,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:42,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:47,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:51,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:12:51,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:12:51,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:12:52,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:12:53,052 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:12:53,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:12:58,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:02,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:03,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:06,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:13:06,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:13:09,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:13,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:15,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:20,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:21,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:21,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:13:21,172 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:13:23,053 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:13:25,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:30,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:31,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:35,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:36,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:13:36,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:13:36,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:13:40,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:41,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:45,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:50,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:51,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:13:51,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:13:51,226 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:13:51,293 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:13:53,064 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:13:56,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:13:59,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:01,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:06,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:14:06,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:14:07,388 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:07,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:12,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:17,452 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:18,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:21,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:14:21,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:14:23,055 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:14:24,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:25,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:29,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:34,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:35,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:36,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:14:36,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:14:40,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:43,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:45,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:50,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:14:51,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:14:51,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:14:51,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:14:51,472 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:14:51,472 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:14:51,475 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:14:51,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:14:53,055 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:14:53,694 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:14:56,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:01,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:01,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:06,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:15:06,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:15:06,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:11,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:11,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:16,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:19,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:21,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:15:21,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:15:22,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:23,055 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:15:28,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:29,938 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:33,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:36,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:15:36,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:15:36,176 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:15:38,934 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:40,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:43,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:47,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:15:49,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:51,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:15:51,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:15:53,056 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:15:55,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:15:55,111 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:00,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:05,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:05,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:06,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:16:06,175 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:16:10,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:13,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:15,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:21,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:16:21,174 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:16:21,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:21,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:23,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:16:27,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:31,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:32,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:37,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:16:37,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:16:37,122 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:16:38,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:39,415 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:43,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:48,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:49,479 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:52,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:16:52,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:16:52,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:16:53,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:16:54,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:16:57,540 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:16:59,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:04,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:07,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:17:07,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:17:07,120 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:17:07,604 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:09,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:15,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:15,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:20,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:22,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:17:22,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:17:22,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:17:23,057 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:17:25,715 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:26,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:31,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:33,768 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:36,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:37,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:17:37,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:17:37,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:17:42,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:43,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:47,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:50,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:17:50,279 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:17:50,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:17:50,282 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:17:50,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:17:51,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:17:52,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:17:52,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:17:52,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:17:52,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:17:53,058 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:17:58,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:01,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:03,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:07,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:18:07,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:18:07,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:18:09,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:09,997 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:14,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:20,058 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:20,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:22,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:18:22,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:18:22,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:18:23,059 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:18:26,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:28,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:31,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:36,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:37,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:18:37,629 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:18:38,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:41,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:46,232 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:47,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:52,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:18:53,061 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:18:53,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:18:53,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:18:53,094 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:18:56,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:18:58,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:03,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:05,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:08,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:19:08,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:19:08,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:19:08,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:13,436 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:13,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:18,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:23,060 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:19:23,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:19:23,092 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:19:23,502 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:24,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:29,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:33,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:35,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:39,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:19:39,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:19:39,443 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:19:40,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:41,616 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:45,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:51,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:51,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:19:53,061 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:19:54,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:19:54,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:19:54,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:19:56,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:19:59,739 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:01,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:06,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:09,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:20:09,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:20:09,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:20:09,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:12,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:17,852 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:18,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:23,064 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:20:24,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:25,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:20:25,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:20:25,045 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:20:27,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:30,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:35,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:37,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:40,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:20:40,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:20:40,045 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:20:40,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:45,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:46,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:50,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:20:53,065 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:20:54,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:20:54,028 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:20:54,029 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:20:54,031 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:20:54,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:20:55,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:20:55,043 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:20:56,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:20:56,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:01,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:06,165 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:06,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:11,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:21:11,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:21:11,110 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:21:12,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:14,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:17,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:22,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:22,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:23,065 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:21:26,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:21:26,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:21:28,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:32,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:33,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:38,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:41,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:21:41,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:21:41,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:21:42,447 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:44,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:50,095 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:50,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:21:53,065 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:21:56,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:21:56,983 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:21:56,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:21:56,986 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:22:00,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:01,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:06,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:09,621 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:11,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:11,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:22:11,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:22:11,985 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:22:17,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:17,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:22,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:23,066 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:22:26,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:22:26,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:22:26,985 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:22:27,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:27,745 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:32,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:37,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:37,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:42,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:43,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:22:43,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:22:43,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:22:45,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:48,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:53,067 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:22:54,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:22:55,945 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:22:58,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:22:58,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:22:58,526 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:22:59,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:04,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:04,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:09,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:13,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:23:13,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:23:13,526 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:23:14,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:15,742 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:20,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:22,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:23,067 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:23:26,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:28,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:23:28,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:23:28,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:23:31,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:32,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:36,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:40,245 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:42,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:43,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:23:43,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:23:48,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:50,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:23:53,068 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:23:54,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:23:57,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:23:57,856 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:23:57,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:23:57,859 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:23:58,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:23:58,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:23:58,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:23:59,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:00,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:04,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:08,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:09,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:14,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:15,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:24:15,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:24:15,298 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:24:18,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:20,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:23,069 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:24:26,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:26,555 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:30,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:24:30,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:24:30,298 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:24:31,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:36,636 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:36,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:41,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:45,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:24:45,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:24:45,298 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:24:46,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:47,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:52,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:24:53,070 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:24:54,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:24:58,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:01,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:25:01,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:25:01,403 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:25:03,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:03,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:08,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:14,079 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:14,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:16,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:25:16,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:25:19,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:22,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:23,070 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:25:25,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:30,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:31,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:25:31,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:25:32,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:35,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:40,882 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:42,279 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:45,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:46,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:25:46,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:25:51,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:25:53,071 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:25:54,360 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:25:57,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:01,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:26:01,404 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:26:03,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:04,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:08,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:12,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:13,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:16,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:26:16,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:26:18,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:22,547 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:23,082 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:26:24,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:29,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:30,610 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:31,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:26:31,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:26:34,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:39,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:40,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:44,765 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:46,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:26:46,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:26:48,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:26:50,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:53,071 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:26:56,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:26:58,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:01,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:27:01,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:27:01,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:05,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:27:05,947 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:27:05,947 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:27:05,949 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:27:06,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:27:06,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:08,929 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:11,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:15,974 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:16,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:27:16,485 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:27:17,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:22,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:23,073 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:27:26,034 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:28,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:32,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:27:32,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:27:32,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:27:33,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:36,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:38,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:44,356 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:44,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:47,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:27:47,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:27:47,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:27:50,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:27:52,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:27:53,073 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:27:56,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:01,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:02,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:02,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:28:02,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:28:02,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:28:07,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:12,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:12,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:17,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:18,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:28:18,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:28:18,311 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:28:20,592 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:23,073 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:28:23,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:28,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:30,663 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:33,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:28:33,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:33,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:28:33,310 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:28:38,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:39,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:44,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:48,308 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:28:48,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:28:48,310 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:28:48,776 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:28:49,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:53,076 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:28:55,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:28:56,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:00,227 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:03,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:29:03,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:29:03,823 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:29:05,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:06,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:10,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:16,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:17,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:18,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:29:18,822 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:29:21,985 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:23,076 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:29:25,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:27,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:32,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:33,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:33,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:29:33,821 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:29:38,033 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:43,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:43,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:48,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:48,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:29:48,824 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:29:48,825 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:29:50,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:29:53,076 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:29:54,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:29:59,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:00,278 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:03,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:30:03,822 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:30:05,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:07,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:30:07,601 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:30:07,602 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:30:07,604 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:30:08,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:30:08,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:10,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:15,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:18,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:18,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:30:18,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:30:18,861 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:30:21,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:23,078 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:30:26,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:26,478 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:31,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:33,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:30:33,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:30:33,863 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:30:36,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:36,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:41,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:46,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:47,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:48,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:30:48,860 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:30:53,078 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:30:53,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:30:54,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:30:58,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:03,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:31:03,860 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:31:03,860 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:31:04,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:04,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:09,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:12,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:14,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:19,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:31:19,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:31:19,921 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:31:20,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:22,845 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:23,079 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:31:26,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:30,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:31,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:34,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:31:34,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:31:34,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:31:37,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:40,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:42,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:47,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:49,029 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:31:49,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:31:49,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:31:49,920 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:31:53,079 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:31:53,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:58,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:31:59,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:03,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:05,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:32:05,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:32:05,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:32:08,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:09,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:13,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:17,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:19,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:20,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:32:20,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:32:20,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:32:23,080 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:32:25,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:27,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:30,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:35,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:32:35,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:32:35,141 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:32:35,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:35,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:40,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:45,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:45,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:50,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:32:50,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:32:50,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:32:50,567 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:32:52,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:32:53,081 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:32:56,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:01,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:02,491 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:05,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:33:05,565 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:33:06,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:09,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:33:09,716 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:33:09,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:33:09,718 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:33:10,542 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:33:10,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:12,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:17,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:20,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:33:20,566 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:33:20,608 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:22,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:23,081 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:33:28,083 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:28,670 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:33,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:35,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:33:35,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:33:35,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:33:38,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:38,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:43,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:46,797 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:33:49,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:50,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:33:50,853 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:33:53,083 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:33:55,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:33:56,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:00,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:04,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:05,852 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:34:05,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:05,853 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:34:11,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:14,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:16,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:20,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:34:20,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:34:20,856 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:34:21,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:23,032 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:23,083 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:34:27,084 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:32,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:33,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:35,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:34:35,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:34:38,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:41,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:34:43,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:48,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:50,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:34:50,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:34:53,084 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:34:54,085 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:34:59,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:04,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:05,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:35:05,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:35:10,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:15,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:20,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:20,853 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:35:20,854 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:35:23,084 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:35:25,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:30,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:31,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:35:35,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:36,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:35:36,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:35:36,440 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:35:38,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:35:40,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:45,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:48,681 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:35:51,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:51,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:35:51,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:35:53,085 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:35:56,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:35:58,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:01,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:06,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:36:06,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:36:06,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:07,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:12,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:16,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:18,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:21,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:36:21,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:36:21,442 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:36:23,086 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:36:24,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:24,927 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:29,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:34,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:34,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:36,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:36:36,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:36:39,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:45,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:45,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:50,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:36:52,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:36:52,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:36:52,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:36:53,087 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:36:53,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:36:53,252 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:36:53,253 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:36:53,256 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:36:54,109 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:36:55,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:36:55,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:00,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:03,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:05,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:07,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:37:07,092 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:37:11,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:13,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:16,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:22,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:37:22,092 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:37:22,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:23,087 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:37:23,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:28,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:33,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:33,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:37,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:37:37,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:37:37,096 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:37:38,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:41,434 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:44,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:49,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:51,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:37:52,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:37:52,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:37:53,088 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:37:55,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:37:59,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:00,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:05,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:07,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:38:07,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:38:07,094 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:38:09,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:10,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:15,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:19,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:20,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:22,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:38:22,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:38:23,090 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:38:25,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:26,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:30,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:35,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:36,914 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:37,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:38:37,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:38:41,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:46,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:46,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:51,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:38:52,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:38:52,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:38:52,732 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:38:53,090 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:38:55,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:38:57,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:02,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:05,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:07,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:07,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:39:07,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:39:12,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:15,189 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:18,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:22,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:39:22,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:39:22,733 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:39:23,093 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:39:24,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:25,370 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:29,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:33,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:34,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:37,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:39:37,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:39:39,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:43,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:45,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:50,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:39:52,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:39:52,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:39:53,091 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:39:53,671 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:39:56,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:01,093 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:05,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:40:05,926 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:40:05,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:40:05,929 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:40:06,765 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:40:06,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:07,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:40:07,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:40:07,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:11,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:17,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:17,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:22,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:23,093 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:40:23,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:40:23,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:40:23,348 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:40:25,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:27,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:32,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:35,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:37,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:38,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:40:38,347 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:40:38,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:40:42,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:43,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:47,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:52,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:40:53,088 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:40:53,093 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:40:53,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:40:53,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:40:58,584 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:03,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:04,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:09,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:09,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:41:09,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:41:09,787 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:41:11,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:14,864 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:20,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:21,267 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:23,093 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:41:24,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:41:24,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:41:25,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:29,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:31,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:36,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:39,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:39,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:41:39,785 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:41:41,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:46,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:49,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:41:52,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:53,094 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:41:55,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:41:55,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:41:55,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:41:57,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:41:57,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:02,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:07,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:08,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:10,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:42:10,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:42:13,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:15,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:18,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:23,096 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:42:24,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:25,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:42:25,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:42:25,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:30,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:33,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:35,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:40,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:42:40,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:42:40,581 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:42:41,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:43,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:46,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:51,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:42:52,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:42:53,095 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:42:55,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:42:55,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:42:55,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:42:57,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:01,956 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:02,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:08,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:09,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:43:09,030 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:43:09,030 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:43:09,033 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:43:10,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:43:10,024 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:10,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:43:10,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:43:10,580 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:43:13,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:19,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:20,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:23,096 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:43:24,097 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:25,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:43:25,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:43:25,957 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:43:28,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:30,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:35,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:38,222 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:40,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:43:40,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:40,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:43:40,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:43:45,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:46,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:51,002 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:43:53,098 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:43:55,341 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:43:55,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:43:55,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:43:55,955 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:43:56,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:01,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:05,410 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:06,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:11,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:11,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:44:11,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:44:11,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:44:13,462 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:16,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:22,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:23,098 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:44:23,521 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:26,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:44:26,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:44:26,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:44:27,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:31,582 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:33,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:38,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:41,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:44:41,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:44:41,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:43,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:48,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:51,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:44:53,098 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:44:54,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:44:56,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:44:56,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:44:56,428 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:44:59,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:01,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:04,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:09,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:09,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:11,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:45:11,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:45:14,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:19,906 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:20,522 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:23,109 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:45:26,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:26,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:45:26,425 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:45:26,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:45:27,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:31,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:36,939 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:38,039 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:41,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:45:41,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:45:42,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:47,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:48,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:52,913 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:45:53,100 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:45:56,376 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:45:56,424 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:45:56,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:45:58,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:03,615 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:06,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:09,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:11,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:46:11,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:46:11,841 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:46:13,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:46:13,046 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:46:13,046 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:46:13,048 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:46:13,518 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:46:13,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:15,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:20,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:23,100 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:46:23,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:25,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:26,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:46:26,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:46:26,840 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:46:31,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:31,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:36,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:41,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:41,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:41,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:46:41,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:46:41,840 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:46:47,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:51,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:46:52,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:53,102 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:46:57,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:46:57,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:46:57,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:46:57,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:46:59,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:02,863 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:08,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:10,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:12,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:47:12,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:47:12,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:47:13,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:18,076 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:19,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:23,103 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:47:25,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:27,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:47:27,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:47:27,620 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:47:28,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:30,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:36,209 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:36,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:41,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:43,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:47:43,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:47:43,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:47:46,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:46,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:51,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:53,103 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:47:54,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:47:57,217 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:47:58,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:47:58,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:47:58,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:48:02,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:04,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:07,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:13,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:48:13,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:48:13,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:13,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:48:14,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:18,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:22,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:23,104 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:48:24,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:29,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:29,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:48:29,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:48:29,392 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:48:32,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:34,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:39,722 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:42,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:44,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:48:44,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:48:44,392 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:48:45,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:49,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:48:50,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:53,104 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:48:56,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:48:59,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:48:59,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:48:59,392 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:48:59,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:01,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:06,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:07,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:12,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:15,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:49:15,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:49:15,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:49:16,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:49:16,401 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:49:16,402 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:49:16,404 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:49:16,888 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:49:17,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:17,894 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:22,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:23,106 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:49:25,950 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:28,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:30,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:49:30,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:49:30,279 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:49:33,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:36,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:39,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:44,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:44,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:45,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:49:45,278 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:49:49,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:49:53,106 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:49:54,127 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:49:55,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:00,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:00,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:50:00,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:50:00,283 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:50:02,174 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:05,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:11,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:12,241 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:15,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:50:15,279 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:50:16,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:20,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:22,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:23,107 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:50:28,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:30,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:50:30,279 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:50:30,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:33,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:38,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:38,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:43,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:45,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:50:45,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:50:45,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:50:48,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:50:49,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:53,108 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:50:55,110 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:50:56,554 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:00,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:00,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:51:00,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:51:00,408 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:51:06,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:06,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:11,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:14,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:15,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:51:15,407 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:51:15,408 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:51:16,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:21,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:23,108 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:51:24,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:27,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:30,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:51:30,410 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:51:30,412 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:51:32,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:32,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:37,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:42,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:43,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:45,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:51:45,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:51:45,409 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:51:48,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:51,960 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:51:53,108 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:51:54,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:51:59,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:00,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:00,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:52:00,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:52:00,410 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:52:05,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:08,079 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:10,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:15,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:52:15,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:52:15,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:52:15,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:16,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:52:16,604 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:52:16,605 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:52:16,607 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:52:17,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:52:18,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:21,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:23,110 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:52:26,198 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:27,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:30,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:52:30,539 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:52:32,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:36,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:37,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:42,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:44,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:45,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:52:45,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:52:48,718 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:53,110 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:52:54,111 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:52:54,380 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:52:59,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:00,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:53:00,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:53:04,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:09,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:14,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:15,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:53:15,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:53:16,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:53:20,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:23,111 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:53:25,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:26,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:53:30,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:53:30,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:53:31,737 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:34,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:53:37,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:42,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:44,698 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:53:45,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:53:45,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:53:47,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:52,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:53,112 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:53:58,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:53:58,783 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:00,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:54:00,540 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:54:03,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:06,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:09,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:14,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:15,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:54:15,541 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:54:16,892 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:19,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:23,112 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:54:24,943 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:25,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:30,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:32,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:54:32,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:54:32,314 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:54:35,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:36,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:41,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:45,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:46,865 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:47,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:54:47,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:54:47,312 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:54:52,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:54:53,113 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:54:55,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:54:58,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:02,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:02,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:55:02,312 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:55:03,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:08,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:10,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:14,561 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:17,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:55:17,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:55:17,514 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:55:20,320 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:20,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:23,114 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:55:26,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:28,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:31,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:32,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:55:32,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:55:32,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:55:36,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:36,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:55:36,993 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:55:36,993 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:55:36,996 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:55:37,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:55:38,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:41,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:46,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:47,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:55:47,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:55:47,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:52,724 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:55:53,115 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:55:56,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:55:58,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:02,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:56:02,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:56:03,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:08,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:13,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:16,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:56:17,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:56:17,512 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:56:18,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:23,116 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:56:24,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:26,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:56:29,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:32,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:56:32,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:56:32,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:56:35,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:36,818 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:56:40,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:44,873 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:56:46,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:47,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:56:47,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:56:47,794 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:56:51,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:56:53,116 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:56:54,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:56:57,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:02,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:57:02,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:57:02,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:02,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:57:02,986 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:08,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:13,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:14,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:18,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:57:18,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:57:18,731 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:57:19,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:21,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:23,117 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:57:25,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:30,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:31,191 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:33,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:57:33,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:57:33,730 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:57:35,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:41,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:41,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:46,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:48,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:57:48,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:57:48,730 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:57:49,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:57:51,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:53,117 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:57:57,869 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:57:59,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:02,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:04,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:58:04,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:58:04,608 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:58:07,417 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:08,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:13,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:16,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:19,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:19,605 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:58:19,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:58:19,606 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:58:23,118 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:58:24,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:26,549 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:29,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:34,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:34,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:58:34,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:58:34,607 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:58:34,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:39,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:44,711 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:45,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:50,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:58:50,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 06:58:50,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:58:50,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:58:51,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 06:58:51,889 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 06:58:51,890 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 06:58:51,892 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 06:58:52,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 06:58:52,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:53,118 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:58:54,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 06:58:56,119 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:01,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:05,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:59:05,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:59:06,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:11,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:16,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:20,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:59:20,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:59:21,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:23,119 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:59:27,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:32,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:35,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:59:35,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:59:37,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:42,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:47,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:50,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 06:59:50,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 06:59:53,119 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 06:59:53,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 06:59:58,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:03,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:05,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:00:05,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:00:08,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:13,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:18,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:20,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:00:20,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:00:23,120 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:00:24,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:29,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:34,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:35,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:00:35,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:00:35,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:00:40,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:45,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:50,041 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:00:50,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:00:50,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:00:53,120 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:00:55,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:00,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:05,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:05,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:01:05,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:01:10,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:15,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:20,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:01:20,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:01:20,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:23,121 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:01:26,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:31,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:35,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:01:35,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:01:36,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:41,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:46,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:50,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:01:50,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:01:52,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:01:53,121 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:01:57,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:02,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:05,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:02:05,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:02:07,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:12,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:17,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:20,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:02:20,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:02:23,122 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:02:23,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:28,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:33,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:35,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:02:35,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:02:38,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:43,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:48,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:50,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:02:50,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:02:53,122 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:02:54,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:02:59,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:04,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:05,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:03:05,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:03:09,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:14,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:19,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:20,750 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:03:20,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:03:23,123 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:03:25,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:30,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:35,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:35,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:03:35,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:03:35,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:03:40,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:45,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:50,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:03:50,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:03:51,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:03:53,124 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:03:57,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:02,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:05,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:04:05,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:04:08,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:13,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:18,010 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:20,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:04:20,751 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:04:23,124 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:04:23,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:28,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:33,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:35,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:04:35,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:04:38,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:43,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:48,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:50,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:04:50,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:04:53,124 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:04:54,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:04:59,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:04,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:05,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:05:05,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:05:10,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:15,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:20,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:20,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:05:20,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:05:23,125 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:05:25,126 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:30,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:35,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:35,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:05:35,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:05:41,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:46,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:50,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:05:50,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:05:51,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:05:53,125 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:05:57,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:02,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:05,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:06:05,752 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:06:07,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:12,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:17,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:20,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:06:20,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:06:22,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:23,126 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:06:28,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:33,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:35,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:06:35,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:06:38,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:43,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:48,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:50,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:06:50,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:06:53,126 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:06:54,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:06:59,135 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:04,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:05,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:07:05,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:07:09,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:14,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:19,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:20,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:07:20,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:07:23,126 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:07:25,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:30,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:35,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:35,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:07:35,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:07:41,011 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:46,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:50,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:07:50,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:07:51,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:07:53,127 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:07:57,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:02,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:05,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:08:05,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:08:07,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:12,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:17,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:20,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:08:20,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:08:23,128 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:08:23,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:28,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:33,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:35,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:08:35,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:08:38,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:43,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:08:43,775 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:08:43,775 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:08:43,778 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:08:44,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:08:44,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:45,486 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:08:49,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:08:50,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:08:50,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:08:53,128 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:08:55,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:00,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:05,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:05,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:09:05,754 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:09:10,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:15,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:17,711 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:09:20,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:09:21,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:09:21,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:09:21,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:23,128 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:09:27,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:32,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:36,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:09:36,237 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:09:37,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:42,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:47,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:51,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:09:51,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:09:51,434 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:09:52,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:09:53,129 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:09:58,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:00,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:10:01,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:10:04,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:08,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:08,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:10:08,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:10:09,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:10,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:10:14,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:15,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:19,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:22,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:23,130 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:10:25,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:29,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:29,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:10:29,576 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:10:30,803 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:31,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:10:35,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:40,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:44,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:44,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:10:44,571 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:10:46,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:51,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:53,130 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:10:57,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:10:59,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:10:59,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:10:59,572 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:11:02,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:07,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:12,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:14,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:11:14,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:11:14,571 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:11:18,817 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:23,131 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:11:24,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:28,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:11:29,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:11:29,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:11:29,571 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:11:29,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:34,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:38,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:11:39,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:44,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:45,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:11:45,801 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:11:45,802 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:11:48,427 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:11:50,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:53,132 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:11:56,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:11:57,481 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:01,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:02,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:12:02,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:12:02,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:12:06,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:09,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:12,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:17,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:12:17,795 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:12:17,796 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:12:18,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:19,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:23,132 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:12:23,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:28,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:29,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:32,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:12:32,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:12:34,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:39,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:39,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:44,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:47,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:12:47,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:12:47,854 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:12:49,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:53,134 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:12:55,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:12:57,918 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:00,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:04,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:13:04,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:13:04,055 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:13:05,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:05,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:10,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:15,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:16,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:19,053 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:13:19,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:13:19,054 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:13:21,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:23,136 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:13:26,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:27,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:32,153 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:34,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:13:34,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:13:34,055 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:13:36,157 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:37,284 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:42,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:46,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:47,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:49,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:13:49,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:13:49,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:13:52,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:13:53,135 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:13:54,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:13:58,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:03,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:04,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:14:04,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:14:04,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:14:04,335 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:09,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:14,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:15,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:20,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:20,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:14:20,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:14:20,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:14:22,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:23,136 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:14:26,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:31,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:32,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:35,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:14:35,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:14:35,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:14:37,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:40,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:14:40,927 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:14:40,927 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:14:40,929 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:14:41,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:14:41,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:42,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:42,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:47,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:50,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:14:50,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:14:50,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:14:52,688 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:14:53,136 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:14:53,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:58,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:14:59,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:03,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:05,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:15:05,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:15:05,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:15:08,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:09,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:14,003 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:19,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:19,875 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:20,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:15:20,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:15:20,333 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:15:23,137 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:15:25,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:27,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:30,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:35,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:36,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:15:36,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:15:36,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:15:37,987 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:40,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:45,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:48,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:15:50,991 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:51,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:15:51,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:15:51,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:15:53,139 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:15:56,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:15:58,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:01,496 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:06,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:16:06,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:16:06,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:08,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:11,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:17,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:18,249 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:21,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:16:21,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:16:22,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:23,139 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:16:27,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:28,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:32,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:36,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:16:36,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:16:36,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:16:38,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:38,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:43,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:48,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:48,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:51,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:16:51,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:16:51,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:16:53,141 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:16:54,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:16:56,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:16:59,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:04,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:06,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:17:06,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:17:06,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:09,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:14,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:14,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:19,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:21,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:17:21,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:17:21,898 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:17:23,140 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:17:24,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:25,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:30,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:34,860 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:35,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:36,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:17:36,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:17:36,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:17:41,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:42,916 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:46,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:50,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:17:50,640 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:17:50,640 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:17:50,643 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:17:50,973 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:17:51,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:17:51,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:17:51,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:17:52,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:17:53,141 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:17:57,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:02,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:02,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:06,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:18:06,898 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:18:08,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:12,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:13,971 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:18,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:20,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:21,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:18:21,895 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:18:23,142 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:18:24,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:30,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:30,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:35,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:37,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:18:37,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:18:37,129 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:18:40,423 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:40,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:45,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:48,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:18:50,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:52,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:18:52,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:18:52,129 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:18:53,142 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:18:56,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:18:58,531 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:01,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:06,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:07,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:07,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:19:07,129 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:19:12,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:16,668 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:17,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:22,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:19:22,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:19:22,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:19:23,153 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:19:23,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:26,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:28,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:33,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:36,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:37,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:19:37,130 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:19:39,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:45,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:46,870 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:19:50,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:52,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:19:52,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:19:52,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:19:53,145 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:19:56,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:19:56,941 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:01,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:04,990 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:06,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:07,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:20:07,815 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:20:12,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:15,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:17,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:22,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:22,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:20:22,815 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:20:23,145 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:20:28,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:29,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:33,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:37,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:20:37,815 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:20:38,178 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:39,036 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:44,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:48,236 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:20:49,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:52,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:20:52,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:20:52,819 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:20:53,147 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:20:55,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:20:56,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:00,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:05,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:21:05,500 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:21:05,501 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:21:05,504 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:21:06,357 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:21:06,358 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:06,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:07,815 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:21:07,816 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:21:08,367 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:12,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:16,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:17,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:22,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:23,149 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:21:23,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:21:23,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:21:23,206 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:21:26,488 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:27,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:32,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:34,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:37,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:38,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:21:38,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:21:42,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:44,613 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:47,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:52,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:21:53,149 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:21:53,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:21:53,204 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:21:53,205 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:21:58,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:02,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:04,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:09,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:22:09,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:22:09,255 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:22:09,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:12,885 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:14,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:19,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:23,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:23,149 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:22:24,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:22:24,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:22:25,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:30,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:31,154 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:35,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:39,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:22:39,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:22:39,249 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:22:40,674 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:41,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:45,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:51,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:22:51,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:22:53,150 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:22:54,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:22:54,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:22:54,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:22:57,470 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:00,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:03,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:08,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:09,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:23:09,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:23:09,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:23:10,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:13,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:18,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:20,496 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:23,150 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:23:24,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:24,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:23:24,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:23:24,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:23:28,543 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:29,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:34,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:38,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:39,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:23:39,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:23:39,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:23:40,419 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:45,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:46,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:23:51,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:53,151 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:23:54,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:23:54,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:23:54,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:23:56,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:23:56,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:01,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:06,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:06,793 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:09,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:24:09,389 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:24:12,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:13,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:24:13,546 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:24:13,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:24:13,550 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:24:13,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:24:14,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:15,842 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:18,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:23,152 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:24:24,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:24,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:24:24,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:24:24,389 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:24:24,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:29,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:33,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:34,916 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:39,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:24:39,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:24:39,758 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:24:40,008 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:43,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:45,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:51,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:24:51,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:24:53,152 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:24:54,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:24:54,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:24:54,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:24:56,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:01,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:02,201 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:07,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:09,756 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:25:09,757 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:25:11,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:12,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:17,989 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:19,334 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:23,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:23,153 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:25:26,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:25:26,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:25:26,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:25:28,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:29,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:33,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:38,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:39,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:41,117 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:25:41,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:25:44,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:47,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:25:49,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:53,154 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:25:55,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:25:56,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:25:56,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:25:56,611 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:00,348 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:04,661 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:05,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:10,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:11,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:26:11,118 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:26:14,723 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:16,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:21,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:23,155 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:26:24,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:26,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:26:26,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:26:26,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:31,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:34,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:36,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:41,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:26:41,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:26:41,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:26:42,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:42,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:47,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:52,409 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:26:52,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:26:53,156 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:26:56,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:26:56,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:26:57,413 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:02,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:03,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:07,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:11,105 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:11,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:27:11,259 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:27:13,478 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:18,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:19,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:27:19,176 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:27:19,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:27:19,178 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:27:20,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:27:20,177 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:21,181 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:23,156 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:27:24,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:27,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:27:27,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:27:27,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:27:30,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:31,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:35,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:40,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:41,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:42,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:27:42,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:27:42,279 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:27:45,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:50,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:51,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:27:53,156 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:27:56,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:27:57,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:27:57,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:27:57,282 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:27:59,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:02,118 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:07,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:09,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:12,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:28:12,279 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:28:12,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:17,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:19,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:22,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:23,157 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:28:27,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:28:27,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:28:27,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:28,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:33,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:37,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:38,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:42,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:28:42,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:28:42,284 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:28:44,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:47,764 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:50,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:53,158 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:28:55,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:28:55,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:28:57,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:28:57,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:29:00,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:04,879 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:06,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:11,373 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:12,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:29:12,281 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:29:12,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:29:12,942 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:16,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:21,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:23,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:23,158 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:29:27,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:27,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:29:27,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:29:31,075 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:32,810 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:37,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:41,143 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:42,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:29:42,280 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:29:43,524 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:48,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:51,202 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:29:53,169 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:29:54,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:29:57,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:29:57,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:29:57,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:29:59,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:01,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:04,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:09,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:10,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:12,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:30:12,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:30:15,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:19,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:21,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:23,159 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:30:26,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:26,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:30:26,260 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:30:26,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:30:26,262 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:30:26,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:30:27,437 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:27,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:30:27,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:30:28,442 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:31,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:37,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:37,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:42,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:43,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:30:43,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:30:43,627 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:30:47,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:47,784 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:52,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:53,160 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:30:55,634 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:30:58,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:30:58,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:30:58,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:30:58,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:31:03,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:05,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:08,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:13,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:31:13,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:31:13,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:31:14,811 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:15,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:19,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:23,161 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:31:23,810 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:25,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:30,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:30,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:31:30,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:31:30,223 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:31:33,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:35,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:40,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:41,934 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:45,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:31:45,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:31:45,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:31:46,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:51,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:31:52,002 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:31:53,161 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:31:57,162 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:00,068 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:00,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:32:00,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:32:00,222 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:32:02,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:08,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:09,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:13,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:15,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:32:15,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:32:15,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:32:19,059 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:19,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:23,163 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:32:24,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:27,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:29,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:30,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:32:30,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:32:30,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:32:35,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:37,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:40,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:45,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:32:45,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:32:45,857 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:32:46,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:47,372 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:51,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:32:53,164 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:32:55,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:32:57,040 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:01,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:33:01,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:33:01,967 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:33:02,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:05,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:07,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:12,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:15,546 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:16,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:33:16,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:33:16,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:33:18,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:23,164 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:33:23,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:25,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:28,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:31,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:33:31,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:33:31,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:33:32,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:33:32,414 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:33:32,414 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:33:32,419 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:33:32,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:33:33,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:33,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:34,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:38,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:43,781 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:44,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:46,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:33:46,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:33:46,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:33:49,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:53,166 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:33:53,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:33:54,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:33:59,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:01,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:01,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:34:01,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:34:05,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:10,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:15,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:15,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:16,964 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:34:16,965 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:34:21,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:23,166 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:34:26,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:27,136 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:32,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:32,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:34:32,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:34:32,191 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:34:36,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:37,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:42,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:44,142 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:47,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:34:47,189 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:34:48,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:53,167 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:34:54,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:34:54,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:34:59,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:02,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:35:02,189 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:35:03,489 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:04,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:09,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:11,538 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:14,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:18,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:35:18,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:35:18,440 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:35:20,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:21,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:23,167 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:35:26,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:29,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:31,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:33,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:35:33,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:35:37,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:39,846 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:42,833 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:48,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:48,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:35:48,439 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:35:49,913 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:53,168 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:35:54,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:35:57,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:35:59,804 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:03,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:36:03,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:36:03,443 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:36:05,643 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:08,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:11,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:16,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:18,092 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:18,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:36:18,440 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:36:22,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:23,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:36:26,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:27,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:32,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:33,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:36:33,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:36:36,196 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:38,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:43,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:43,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:36:43,173 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:36:43,173 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:36:43,175 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:36:43,251 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:36:45,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:46,268 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:48,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:48,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:36:48,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:36:53,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:36:54,170 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:36:54,315 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:36:59,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:03,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:37:03,441 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:37:04,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:04,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:09,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:12,435 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:14,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:18,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:37:18,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:37:18,846 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:37:19,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:21,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:23,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:37:25,171 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:30,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:31,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:33,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:37:33,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:37:33,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:37:36,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:39,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:41,453 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:46,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:48,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:37:48,844 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:37:48,845 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:37:49,763 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:37:52,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:53,170 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:37:57,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:37:57,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:02,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:04,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:38:04,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:38:04,179 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:38:07,883 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:08,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:13,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:15,944 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:18,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:19,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:38:19,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:38:23,172 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:38:23,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:26,020 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:28,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:33,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:34,080 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:34,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:38:34,177 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:38:39,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:44,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:44,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:49,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:38:49,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:38:49,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:38:49,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:38:52,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:38:53,172 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:38:55,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:00,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:02,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:04,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:39:04,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:39:04,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:39:06,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:11,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:12,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:16,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:19,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:39:19,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:39:19,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:39:22,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:22,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:23,173 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:39:27,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:30,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:32,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:34,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:39:34,876 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:39:34,877 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:39:38,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:40,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:43,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:48,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:39:48,076 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:39:48,076 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:39:48,079 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:39:48,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:39:49,080 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:49,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:49,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:39:49,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:39:50,645 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:39:53,173 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:39:54,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:59,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:39:59,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:04,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:05,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:40:05,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:40:05,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:40:07,766 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:09,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:14,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:17,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:20,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:20,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:40:20,656 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:40:23,175 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:40:25,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:25,893 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:30,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:35,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:40:35,656 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:40:35,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:35,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:40,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:44,031 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:46,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:51,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:40:51,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:40:51,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:40:51,789 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:40:53,175 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:40:54,098 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:40:57,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:02,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:04,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:06,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:41:06,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:41:06,788 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:41:08,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:12,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:13,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:18,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:21,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:41:21,788 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:41:22,290 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:23,175 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:41:24,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:29,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:30,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:34,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:37,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:41:37,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:41:37,609 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:41:39,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:40,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:44,752 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:50,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:50,495 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:41:52,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:41:52,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:41:52,607 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:41:53,176 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:41:56,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:41:58,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:01,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:07,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:07,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:42:07,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:42:07,607 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:42:08,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:12,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:18,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:18,673 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:22,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:42:22,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:42:22,611 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:42:23,190 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:42:23,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:28,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:28,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:33,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:36,799 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:37,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:42:37,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:42:37,608 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:42:38,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:43,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:46,867 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:48,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:53,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:42:53,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:42:53,064 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:42:53,235 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:42:54,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:42:54,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:42:54,266 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:42:54,267 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:42:54,268 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:42:54,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:42:55,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:56,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:42:59,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:03,967 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:05,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:08,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:43:08,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:43:08,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:43:10,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:14,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:16,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:21,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:22,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:23,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:43:23,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:43:23,302 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:43:26,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:31,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:32,155 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:36,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:38,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:43:38,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:43:38,719 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:43:41,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:42,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:46,763 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:50,287 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:43:52,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:43:53,180 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:43:53,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:43:53,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:43:57,901 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:00,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:03,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:08,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:08,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:44:08,717 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:44:10,426 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:13,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:18,471 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:19,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:23,181 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:44:24,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:44:24,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:44:24,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:44:24,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:26,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:30,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:36,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:36,587 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:39,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:44:39,426 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:44:41,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:46,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:47,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:52,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:44:53,181 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:44:54,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:44:54,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:44:54,427 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:44:54,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:44:57,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:03,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:04,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:08,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:10,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:45:10,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:45:10,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:45:12,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:13,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:18,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:22,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:23,182 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:45:24,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:25,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:45:25,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:45:25,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:45:29,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:32,972 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:34,839 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:40,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:45:40,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:45:40,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:45:40,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:41,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:45,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:51,055 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:51,101 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:45:53,182 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:45:56,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:45:56,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:45:56,144 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:45:56,145 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:45:57,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:45:57,286 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:45:57,286 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:45:57,288 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:45:58,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:45:59,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:00,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:01,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:07,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:08,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:11,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:46:11,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:46:11,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:46:12,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:16,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:17,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:22,438 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:23,183 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:46:26,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:46:26,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:46:26,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:46:26,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:27,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:32,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:36,421 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:38,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:42,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:46:42,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:46:42,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:46:44,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:44,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:49,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:53,184 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:46:54,545 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:46:55,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:46:57,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:46:57,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:46:57,131 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:47:00,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:02,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:05,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:10,447 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:12,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:47:12,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:47:12,131 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:47:12,669 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:16,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:21,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:22,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:23,184 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:47:27,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:28,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:47:28,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:47:28,294 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:47:30,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:32,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:37,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:40,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:42,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:43,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:47:43,292 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:47:48,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:50,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:47:53,116 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:53,185 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:47:58,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:47:58,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:47:58,291 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:48:03,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:05,023 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:08,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:13,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:13,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:48:13,292 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:48:13,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:18,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:23,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:23,185 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:48:24,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:29,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:48:29,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:48:29,014 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:48:29,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:33,221 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:34,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:39,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:43,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:44,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:48:44,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:48:44,013 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:48:45,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:50,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:51,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:48:53,186 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:48:56,187 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:48:59,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:48:59,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:48:59,015 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:49:00,414 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:01,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:06,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:08,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:49:08,696 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:49:08,697 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:49:08,701 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:49:09,467 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:49:09,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:10,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:11,703 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:14,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:49:14,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:49:14,013 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:49:17,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:20,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:23,187 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:49:23,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:28,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:29,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:49:29,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:49:29,014 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:49:29,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:34,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:38,664 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:39,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:44,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:44,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:49:44,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:49:44,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:49:46,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:49,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:53,188 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:49:55,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:49:56,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:49:59,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:49:59,686 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:50:00,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:06,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:06,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:11,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:14,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:50:14,687 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:50:14,896 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:16,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:21,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:23,188 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:50:24,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:27,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:30,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:50:30,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:50:30,209 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:50:32,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:33,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:37,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:42,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:43,083 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:45,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:50:45,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:50:45,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:50:48,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:51,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:50:53,189 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:50:54,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:50:59,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:00,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:51:00,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:51:00,207 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:51:01,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:05,412 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:09,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:11,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:15,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:51:15,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:51:15,782 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:51:16,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:19,366 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:21,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:23,191 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:51:27,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:27,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:30,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:51:30,780 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:51:32,990 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:37,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:38,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:43,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:45,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:51:45,780 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:51:47,569 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:49,009 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:53,193 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:51:54,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:51:55,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:51:59,879 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:01,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:52:01,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:52:01,723 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:52:04,685 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:05,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:10,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:11,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:52:11,701 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:52:11,701 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:52:11,705 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:52:11,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:52:12,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:13,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:16,707 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:16,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:52:16,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:52:16,721 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:52:22,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:22,813 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:23,193 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:52:28,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:30,864 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:31,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:52:31,721 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:52:33,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:38,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:40,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:44,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:47,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:52:47,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:52:47,216 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:52:50,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:51,010 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:52:53,194 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:52:55,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:52:59,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:01,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:02,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:53:02,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:53:06,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:09,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:12,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:17,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:17,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:53:17,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:17,215 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:53:22,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:23,195 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:53:27,253 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:27,838 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:32,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:33,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:53:33,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:53:33,730 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:53:37,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:37,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:42,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:47,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:48,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:48,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:53:48,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:53:48,729 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:53:53,195 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:53:54,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:53:55,445 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:53:59,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:03,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:03,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:54:03,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:54:03,730 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:54:04,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:10,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:13,566 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:15,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:19,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:54:19,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:54:19,938 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:54:20,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:23,196 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:54:23,625 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:26,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:31,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:33,696 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:34,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:54:34,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:54:34,938 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:54:37,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:41,760 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:42,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:47,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:49,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:54:49,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:54:49,937 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:54:51,824 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:54:53,154 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:53,207 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:54:58,968 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:54:59,877 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:03,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:05,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:55:05,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:55:05,852 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:55:08,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:09,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:14,883 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:15,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:55:15,772 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:55:15,772 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:55:15,775 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:55:15,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:55:16,999 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:18,005 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:20,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:20,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:55:20,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:55:20,851 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:55:23,198 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:55:26,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:27,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:31,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:35,123 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:35,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:55:35,851 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:55:37,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:42,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:45,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:47,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:51,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:55:51,230 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:55:51,232 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:55:53,198 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:55:53,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:55:53,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:55:58,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:03,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:03,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:06,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:56:06,230 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:56:08,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:13,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:13,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:18,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:21,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:56:21,230 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:56:21,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:23,198 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:56:24,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:29,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:31,463 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:34,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:36,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:56:36,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:56:36,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:56:39,519 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:40,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:45,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:49,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:56:51,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:51,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:56:51,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:56:51,873 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:56:53,199 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:56:56,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:56:57,638 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:01,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:06,873 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:57:06,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:57:06,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:06,875 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:57:07,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:12,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:17,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:17,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:22,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:23,200 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:57:23,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:57:23,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:57:23,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:57:25,841 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:27,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:32,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:35,914 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:38,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:38,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:57:38,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:57:38,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:57:43,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:45,975 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:48,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:53,200 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:57:53,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:57:53,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:57:53,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:57:53,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:57:56,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:57:58,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:03,613 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:08,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:09,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:58:09,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:58:13,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:18,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:21,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:58:23,201 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:58:24,202 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:24,482 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:58:24,483 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:58:29,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:31,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:58:34,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:38,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 07:58:38,590 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 07:58:38,590 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 07:58:38,593 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 07:58:39,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 07:58:39,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:58:39,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:58:39,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:58:39,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:40,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:58:44,683 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:49,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:58:50,514 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:53,201 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:58:54,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:58:54,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:58:55,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:58:59,444 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:00,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:05,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:09,483 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:59:09,484 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:59:09,522 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:11,647 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:17,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:17,584 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:22,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:23,203 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:59:24,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 07:59:24,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:59:24,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:59:27,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:27,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:32,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:35,697 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:37,957 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:39,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:59:39,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:59:43,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:45,754 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:49,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:53,203 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 07:59:54,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 07:59:54,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 07:59:54,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 07:59:55,812 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 07:59:59,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:03,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:05,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:10,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:11,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:00:11,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:00:11,094 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:00:13,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:15,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:20,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:23,204 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:00:23,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:26,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:00:26,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:00:26,092 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:26,092 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:00:32,060 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:32,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:37,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:41,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:00:41,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:00:41,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:00:42,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:42,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:47,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:52,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:00:52,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:00:53,204 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:00:56,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:00:56,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:00:56,100 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:00:58,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:02,261 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:03,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:08,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:11,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:01:11,100 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:01:11,101 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:01:12,319 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:14,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:19,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:20,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:23,215 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:01:25,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:26,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:01:26,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:01:29,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:30,302 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:35,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:37,504 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:40,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:41,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:01:41,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:01:45,504 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:45,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:01:45,571 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:01:45,571 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:01:45,574 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:01:46,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:01:46,564 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:47,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:50,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:53,206 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:01:55,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:01:56,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:01:56,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:01:56,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:02:01,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:06,477 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:09,708 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:02:11,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:02:11,258 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:02:11,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:16,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:21,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:23,206 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:02:27,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:02:27,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:02:27,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:02:27,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:32,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:37,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:42,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:02:42,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:02:43,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:48,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:53,207 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:02:53,993 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:02:54,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:02:57,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:02:57,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:02:59,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:02,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:05,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:10,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:12,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:03:12,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:03:12,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:03:12,112 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:15,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:21,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:22,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:23,208 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:03:26,209 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:27,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:03:27,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:03:27,935 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:03:30,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:31,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:36,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:40,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:41,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:42,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:03:42,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:03:47,181 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:50,394 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:03:52,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:53,208 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:03:57,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:03:57,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:03:57,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:03:58,457 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:03,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:08,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:08,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:12,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:04:12,931 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:04:14,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:18,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:20,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:23,209 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:04:25,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:25,970 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:27,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:04:27,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:04:31,189 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:35,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:37,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:42,134 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:42,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:04:42,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:04:42,934 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:04:45,761 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:47,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:52,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:53,209 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:04:53,819 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:04:57,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:04:57,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:04:57,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:05:02,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:03,880 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:07,984 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:11,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:12,931 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:05:12,932 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:05:13,178 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:18,180 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:21,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:23,210 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:05:23,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:28,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:28,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:05:28,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:05:28,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:05:29,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:05:29,658 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:05:29,658 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:05:29,661 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:05:30,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:05:31,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:32,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:33,663 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:38,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:40,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:43,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:05:43,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:05:43,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:48,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:50,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:53,210 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:05:54,211 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:05:58,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:05:58,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:05:58,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:05:59,783 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:04,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:08,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:10,158 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:13,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:06:13,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:06:13,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:06:16,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:16,420 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:21,109 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:23,212 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:06:26,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:26,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:28,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:06:28,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:06:28,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:06:32,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:36,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:37,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:42,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:43,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:06:43,951 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:06:44,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:48,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:53,212 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:06:53,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:06:54,649 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:06:58,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:00,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:07:00,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:07:00,503 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:07:02,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:03,639 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:08,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:12,803 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:14,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:15,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:07:15,502 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:07:19,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:22,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:23,213 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:07:25,214 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:29,923 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:30,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:07:30,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:07:30,502 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:07:30,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:35,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:37,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:41,711 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:45,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:07:45,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:07:45,583 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:07:46,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:48,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:07:51,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:53,214 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:07:57,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:07:58,131 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:00,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:08:00,581 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:08:02,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:06,185 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:08,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:13,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:15,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:08:15,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:08:15,596 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:08:16,303 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:18,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:23,214 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:08:23,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:26,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:28,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:31,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:08:31,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:08:31,949 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:08:33,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:08:33,167 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:08:33,168 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:08:33,170 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:08:33,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:08:34,172 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:34,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:35,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:39,173 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:44,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:44,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:46,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:08:46,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:08:46,948 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:08:50,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:08:52,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:08:53,215 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:08:55,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:01,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:01,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:09:01,947 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:09:02,684 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:06,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:10,736 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:11,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:16,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:17,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:09:17,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:09:17,359 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:09:20,804 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:22,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:23,216 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:09:27,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:28,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:32,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:09:32,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:09:32,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:09:32,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:38,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:38,922 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:43,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:46,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:47,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:09:47,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:09:47,357 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:09:48,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:53,217 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:09:54,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:09:57,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:09:59,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:03,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:10:03,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:10:03,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:10:04,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:07,108 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:09,933 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:14,161 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:15,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:18,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:10:18,586 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:10:21,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:23,217 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:10:24,219 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:27,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:32,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:33,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:10:33,586 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:10:34,294 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:37,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:42,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:43,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:48,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:48,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:10:48,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:10:48,590 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:10:52,412 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:10:53,218 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:10:54,218 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:10:59,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:02,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:03,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:11:03,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:11:04,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:10,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:10,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:15,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:18,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:11:18,588 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:11:20,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:20,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:23,218 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:11:26,219 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:28,646 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:31,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:33,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:11:33,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:11:36,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:37,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:11:37,055 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:11:37,056 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:11:37,059 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:11:37,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:11:38,703 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:39,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:42,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:47,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:48,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:11:48,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:11:48,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:52,827 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:11:53,220 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:11:56,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:11:58,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:03,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:03,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:12:03,588 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:12:04,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:08,212 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:13,213 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:14,968 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:18,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:12:18,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:12:18,816 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:23,220 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:12:24,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:25,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:29,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:33,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:12:33,587 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:12:34,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:35,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:39,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:43,179 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:45,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:49,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:12:49,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:12:49,370 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:12:50,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:12:53,220 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:12:53,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:12:56,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:01,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:03,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:04,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:13:04,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:13:06,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:11,364 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:12,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:17,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:19,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:13:19,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:13:19,368 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:13:20,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:22,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:23,221 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:13:28,222 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:30,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:33,501 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:34,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:13:34,366 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:13:38,563 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:39,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:44,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:48,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:13:49,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:13:49,375 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:13:49,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:53,222 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:13:55,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:13:56,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:00,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:04,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:14:04,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:14:05,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:06,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:10,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:14,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:16,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:19,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:14:19,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:14:21,585 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:23,233 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:14:24,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:27,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:32,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:32,902 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:34,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:14:34,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:14:37,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:41,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:14:41,820 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:14:41,821 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:14:41,826 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:14:41,964 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:14:42,828 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:42,971 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:43,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:47,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:49,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:14:49,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:14:49,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:14:53,038 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:14:53,223 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:14:53,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:14:58,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:01,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:03,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:04,629 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:15:04,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:15:04,630 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:15:08,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:11,210 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:13,986 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:19,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:19,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:15:19,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:15:19,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:19,631 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:15:23,223 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:15:25,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:29,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:30,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:35,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:15:35,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:15:35,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:15:35,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:39,458 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:41,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:46,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:47,508 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:50,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:15:50,397 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:15:52,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:15:53,225 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:15:57,568 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:15:57,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:02,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:05,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:16:05,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:16:05,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:16:08,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:10,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:13,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:18,704 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:18,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:20,397 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:16:20,398 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:16:23,225 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:16:24,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:28,774 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:29,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:34,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:35,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:16:35,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:16:35,706 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:16:38,837 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:39,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:44,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:46,897 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:49,847 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:50,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:16:50,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:16:50,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:16:53,226 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:16:54,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:16:56,966 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:16:59,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:05,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:05,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:05,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:17:05,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:17:05,705 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:17:10,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:15,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:16,057 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:21,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:21,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:17:21,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:17:21,082 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:17:23,141 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:23,227 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:17:26,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:31,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:33,211 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:36,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:17:36,079 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:17:36,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:41,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:42,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:47,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:49,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:17:49,813 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:17:49,814 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:17:49,816 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:17:50,327 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:17:51,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:17:51,079 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:17:51,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:52,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:52,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:17:53,227 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:17:58,229 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:17:59,385 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:03,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:06,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:18:06,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:18:06,652 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:18:09,442 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:09,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:14,699 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:19,517 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:20,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:21,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:18:21,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:18:21,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:18:23,230 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:18:25,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:27,570 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:31,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:36,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:36,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:18:36,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:18:36,651 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:18:37,636 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:41,848 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:45,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:47,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:52,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:18:52,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:18:52,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:18:52,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:18:53,230 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:18:55,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:18:58,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:03,073 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:05,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:07,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:19:07,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:19:08,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:13,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:14,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:18,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:22,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:22,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:19:22,952 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:19:23,230 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:19:24,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:29,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:30,985 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:34,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:38,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:19:38,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:19:38,467 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:19:39,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:41,047 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:44,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:49,100 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:19:50,484 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:53,231 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:19:53,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:19:53,465 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:19:55,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:19:59,167 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:00,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:05,696 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:07,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:08,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:20:08,465 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:20:11,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:17,273 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:17,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:22,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:23,232 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:20:23,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:20:23,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:20:23,468 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:20:25,328 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:28,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:34,426 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:35,399 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:38,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:20:38,466 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:20:39,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:43,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:44,875 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:49,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:20:51,932 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:20:51,933 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:20:51,934 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:20:51,936 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:20:52,511 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:20:53,232 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:20:53,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:20:53,466 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:20:53,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:54,521 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:20:55,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:00,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:01,572 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:05,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:08,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:21:08,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:21:08,688 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:21:11,631 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:11,868 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:16,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:21,692 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:22,044 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:23,232 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:21:23,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:21:23,685 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:21:27,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:29,748 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:32,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:38,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:38,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:21:38,685 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:21:39,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:43,843 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:47,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:21:49,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:53,234 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:21:54,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:21:54,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:21:54,372 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:21:54,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:21:57,926 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:00,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:05,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:05,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:09,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:22:09,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:22:09,372 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:22:10,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:15,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:16,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:20,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:23,234 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:22:24,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:24,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:22:24,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:22:24,371 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:22:25,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:30,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:33,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:36,147 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:40,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:22:40,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:22:40,049 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:22:42,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:43,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:47,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:51,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:22:52,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:22:53,234 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:22:55,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:22:55,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:22:55,048 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:22:58,207 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:01,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:04,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:09,393 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:09,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:10,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:23:10,047 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:23:15,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:19,455 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:20,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:23,235 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:23:26,236 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:26,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:23:26,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:23:26,537 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:23:29,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:31,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:36,872 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:37,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:41,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:23:41,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:23:41,535 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:23:42,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:47,644 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:23:48,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:53,050 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:53,235 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:23:57,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:23:57,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:23:57,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:23:58,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:23:58,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:23:58,793 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:23:58,793 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:23:58,796 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:23:59,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:23:59,719 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:00,724 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:03,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:08,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:11,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:12,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:24:12,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:24:12,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:24:14,589 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:19,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:21,861 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:23,236 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:24:25,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:27,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:24:27,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:24:27,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:24:30,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:33,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:36,086 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:41,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:42,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:24:42,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:24:42,386 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:24:44,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:46,017 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:46,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:51,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:53,236 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:24:54,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:24:57,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:24:57,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:24:57,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:24:57,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:25:03,199 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:04,130 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:08,200 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:12,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:25:12,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:25:12,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:25:13,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:14,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:18,581 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:23,237 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:25:24,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:24,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:27,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:25:27,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:25:27,387 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:25:29,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:34,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:34,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:39,892 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:42,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:25:42,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:25:42,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:25:43,402 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:45,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:50,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:51,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:25:53,237 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:25:56,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:25:57,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:25:57,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:25:57,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:26:01,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:01,849 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:06,850 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:11,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:12,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:26:12,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:12,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:26:12,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:26:17,668 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:21,689 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:23,238 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:26:23,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:28,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:28,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:26:28,466 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:26:28,468 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:26:31,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:33,552 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:38,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:39,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:43,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:26:43,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:26:43,466 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:26:43,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:49,660 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:49,947 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:26:53,238 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:26:55,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:26:58,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:26:58,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:27:00,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:00,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:05,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:10,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:11,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:13,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:27:13,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:27:16,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:17,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:27:17,749 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:27:17,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:27:17,752 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:27:18,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:27:20,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:22,755 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:23,239 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:27:28,201 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:28,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:28,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:27:28,476 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:27:33,725 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:38,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:39,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:43,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:27:43,481 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:27:44,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:48,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:50,603 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:53,241 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:27:56,244 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:27:58,382 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:27:58,476 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:27:58,477 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:28:01,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:06,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:08,446 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:11,705 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:14,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:28:14,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:28:14,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:28:16,525 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:17,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:22,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:23,243 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:28:26,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:28,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:29,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:28:29,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:28:29,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:28:33,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:35,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:38,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:43,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:44,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:28:44,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:28:44,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:44,102 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:28:49,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:53,244 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:28:53,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:28:55,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:28:59,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:28:59,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:28:59,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:29:00,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:03,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:05,753 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:10,754 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:11,900 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:14,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:29:14,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:29:16,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:21,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:21,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:23,245 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:29:27,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:29,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:29:29,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:29:29,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:29:30,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:32,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:37,408 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:40,070 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:42,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:44,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:29:44,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:29:48,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:50,145 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:29:53,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:53,245 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:29:58,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:29:59,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:29:59,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:30:03,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:04,226 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:08,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:13,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:14,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:30:14,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:30:14,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:19,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:22,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:23,246 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:30:25,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:29,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:30:29,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:30:29,369 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:30:30,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:30,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:30:30,925 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:30:30,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:30:30,936 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:30:31,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:30:32,418 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:35,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:41,467 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:42,493 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:44,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:30:44,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:30:44,367 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:30:46,617 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:51,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:52,561 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:30:53,247 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:30:57,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:30:59,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:30:59,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:30:59,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:31:00,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:02,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:07,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:10,674 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:13,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:14,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:31:14,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:31:14,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:31:18,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:20,740 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:23,210 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:23,247 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:31:28,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:28,794 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:29,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:31:29,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:31:29,377 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:31:33,627 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:37,859 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:39,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:44,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:31:44,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:31:44,380 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:31:44,632 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:47,912 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:50,406 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:53,248 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:31:55,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:31:55,962 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:31:59,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:31:59,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:31:59,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:32:01,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:06,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:06,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:11,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:14,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:32:14,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:32:14,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:32:16,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:17,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:22,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:23,249 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:32:24,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:28,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:29,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:32:29,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:32:29,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:32:33,894 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:34,443 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:38,895 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:44,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:44,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:32:44,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:32:44,378 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:32:44,578 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:49,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:32:53,249 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:32:54,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:32:55,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:00,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:00,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:33:00,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:33:00,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:33:04,714 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:05,896 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:10,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:14,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:15,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:33:15,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:33:15,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:33:16,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:22,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:22,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:23,249 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:33:27,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:30,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:33:30,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:33:30,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:33:32,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:32,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:37,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:40,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:33:40,302 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:33:40,303 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:33:40,305 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:33:40,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:33:42,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:43,307 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:45,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:33:45,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:33:45,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:33:48,486 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:52,035 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:33:53,250 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:33:54,251 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:33:59,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:00,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:34:00,322 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:34:02,099 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:04,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:10,164 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:10,232 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:15,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:15,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:34:15,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:34:15,325 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:34:20,282 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:20,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:23,250 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:34:26,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:30,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:34:30,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:34:30,343 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:31,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:36,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:38,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:42,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:45,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:34:45,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:34:47,156 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:48,501 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:52,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:34:53,252 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:34:56,560 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:34:57,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:00,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:35:00,323 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:35:03,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:06,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:09,235 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:15,208 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:15,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:35:15,324 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:35:16,690 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:20,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:23,252 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:35:24,746 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:26,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:31,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:35:31,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:35:31,072 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:35:32,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:34,808 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:37,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:42,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:42,982 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:46,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:35:46,070 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:35:48,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:35:52,952 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:35:53,253 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:35:54,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:00,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:01,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:01,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:36:01,071 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:36:05,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:11,074 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:11,184 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:16,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:36:16,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:36:16,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:36:16,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:19,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:22,197 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:23,253 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:36:27,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:29,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:31,087 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:36:31,088 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:36:33,253 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:38,280 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:38,461 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:43,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:46,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:36:46,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:36:46,090 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:36:46,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:36:46,797 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:36:46,798 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:36:46,801 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:36:47,353 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:36:48,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:48,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:49,351 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:53,254 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:36:54,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:36:58,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:36:59,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:01,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:37:01,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:37:05,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:08,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:10,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:16,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:37:16,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:37:16,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:16,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:21,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:23,256 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:37:26,589 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:26,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:31,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:37:31,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:37:31,093 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:37:32,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:34,640 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:38,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:43,027 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:44,713 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:46,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:37:46,089 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:37:48,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:53,256 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:37:54,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:37:54,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:37:59,268 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:01,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:38:01,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:38:01,390 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:38:04,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:04,839 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:09,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:12,895 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:14,661 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:16,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:38:16,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:38:16,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:38:20,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:22,958 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:23,256 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:38:26,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:31,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:31,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:38:31,388 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:38:31,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:36,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:41,087 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:41,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:46,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:47,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:38:47,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:38:47,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:38:51,159 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:38:52,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:53,257 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:38:58,137 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:38:59,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:02,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:39:02,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:39:02,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:39:03,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:09,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:09,295 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:14,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:17,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:39:17,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:39:17,355 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:19,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:23,270 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:39:25,090 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:27,419 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:30,091 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:32,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:39:32,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:39:32,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:39:35,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:35,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:40,823 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:45,529 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:46,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:47,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:39:47,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:39:47,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:39:51,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:39:52,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:39:52,327 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:39:52,327 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:39:52,329 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:39:52,567 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:39:53,258 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:39:54,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:55,581 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:39:57,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:02,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:02,630 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:02,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:40:02,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:40:02,682 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:40:07,940 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:12,701 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:13,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:18,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:18,636 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:40:18,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:40:18,639 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:40:20,758 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:23,259 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:40:24,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:29,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:30,816 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:33,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:40:33,638 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:40:33,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:40:34,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:40,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:40,872 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:45,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:48,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:40:48,638 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:40:48,926 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:40:50,857 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:53,259 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:40:56,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:40:58,984 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:01,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:04,921 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:41:04,924 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:41:04,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:41:07,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:07,045 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:12,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:17,228 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:17,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:19,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:41:19,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:41:19,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:41:23,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:23,260 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:41:27,291 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:28,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:33,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:34,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:41:34,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:41:34,923 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:41:35,339 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:39,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:44,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:45,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:49,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:41:50,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:41:50,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:41:50,968 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:41:53,260 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:41:53,470 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:41:55,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:00,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:03,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:05,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:42:05,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:05,966 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:42:05,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:42:11,190 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:11,594 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:16,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:20,965 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:42:20,966 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:42:21,716 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:22,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:23,261 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:42:27,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:28,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:33,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:36,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:42:36,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:42:36,900 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:42:38,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:38,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:43,071 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:48,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:48,881 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:51,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:42:51,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:42:51,926 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:42:53,261 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:42:54,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:42:56,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:42:56,906 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:42:56,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:42:56,908 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:42:57,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:42:59,066 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:42:59,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:04,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:06,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:43:06,900 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:43:06,900 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:43:07,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:43:10,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:15,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:17,205 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:43:20,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:22,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:43:22,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:43:22,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:43:23,262 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:43:25,263 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:43:25,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:30,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:35,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:43:36,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:37,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:43:37,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:43:37,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:43:41,781 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:45,401 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:43:47,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:52,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:43:52,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:43:52,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:43:53,262 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:43:58,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:03,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:07,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:44:07,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:44:07,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:44:08,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:13,788 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:18,789 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:22,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:44:22,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:44:23,263 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:44:23,858 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:44:24,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:29,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:32,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:44:34,858 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:37,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:44:37,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:44:39,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:42,976 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:44:45,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:50,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:44:52,625 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:44:52,626 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:44:53,037 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:44:53,263 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:44:56,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:01,902 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:03,104 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:06,903 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:45:08,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:45:08,624 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:45:11,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:12,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:17,782 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:21,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:23,263 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:45:23,265 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:45:23,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:45:23,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:45:28,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:29,272 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:33,709 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:45:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:45:38,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:45:38,862 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:39,346 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:43,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:49,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:49,408 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:45:53,277 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:45:53,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:45:53,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:45:53,623 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:45:54,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:45:57,461 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:00,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:05,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:07,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:46:08,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:46:08,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:46:10,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:16,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:17,595 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:21,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:23,266 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:46:23,623 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:46:23,624 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:46:23,625 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:46:25,648 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:27,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:32,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:33,234 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:46:33,235 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:46:33,236 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:46:33,238 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:46:33,717 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:46:35,727 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:38,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:38,621 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:46:38,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:46:43,837 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:45,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:48,841 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:53,266 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:46:53,622 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:46:53,622 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:46:53,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:46:54,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:46:59,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:01,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:05,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:09,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:47:09,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:47:09,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:47:11,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:11,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:16,401 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:21,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:22,043 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:23,267 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:47:24,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:47:24,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:47:24,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:47:26,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:31,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:32,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:36,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:39,168 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:39,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:47:39,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:47:42,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:47,994 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:49,227 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:47:52,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:53,267 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:47:55,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:47:55,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:47:55,603 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:47:58,762 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:47:59,279 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:03,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:07,330 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:08,786 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:10,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:48:10,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:48:10,600 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:48:13,787 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:15,383 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:19,691 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:23,268 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:48:25,451 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:25,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:48:25,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:48:25,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:25,601 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:48:30,854 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:35,514 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:36,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:40,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:48:40,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:48:40,604 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:48:42,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:43,562 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:47,799 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:53,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:48:53,268 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:48:53,626 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:48:55,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:48:55,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:48:55,602 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:48:58,845 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:01,680 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:04,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:10,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:10,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:49:10,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:49:10,601 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:49:11,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:15,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:19,805 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:21,414 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:23,269 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:49:26,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:49:26,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:49:26,385 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:49:26,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:29,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:31,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:36,607 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:36,751 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:49:36,752 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:49:36,753 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:49:36,756 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:49:36,925 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:49:37,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:41,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:49:41,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:49:42,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:47,712 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:48,000 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:52,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:49:53,270 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:49:56,062 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:49:56,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:49:56,383 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:49:58,633 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:03,634 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:06,129 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:50:08,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:12,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:50:12,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:50:12,551 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:50:13,732 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:16,190 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:50:18,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:23,270 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:50:24,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:50:24,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:27,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:50:27,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:50:29,818 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:34,310 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:50:35,074 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:40,075 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:42,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:50:42,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:50:44,381 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:50:45,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:50,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:53,271 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:50:56,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:50:57,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:50:57,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:51:01,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:06,715 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:11,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:12,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:51:12,549 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:51:16,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:21,790 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:51:21,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:23,271 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:51:27,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:27,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:51:27,690 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:51:27,692 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:51:29,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:51:33,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:38,007 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:39,924 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:51:42,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:51:42,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:51:43,906 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:48,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:53,281 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:51:54,150 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:51:54,282 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:51:57,688 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:51:57,689 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:51:59,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:04,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:09,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:13,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:52:13,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:52:13,797 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:52:15,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:18,286 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:52:20,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:23,273 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:52:25,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:26,338 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:52:28,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:52:28,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:52:28,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:52:30,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:36,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:36,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:52:41,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:43,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:52:43,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:52:43,747 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:52:44,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:52:46,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:51,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:53,273 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:52:54,516 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:52:57,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:52:58,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:52:58,749 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:52:58,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:53:02,716 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:04,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:07,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:13,500 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:13,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:13,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:53:13,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:53:13,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:53:18,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:21,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:23,274 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:53:24,275 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:28,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:53:28,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:53:28,750 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:53:29,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:29,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:53:29,946 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:53:29,946 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:53:29,949 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:53:30,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:53:31,780 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:34,951 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:40,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:41,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:43,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:53:43,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:53:43,748 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:53:45,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:49,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:53:51,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:53,274 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:53:57,276 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:53:58,747 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:53:58,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:53:58,749 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:53:59,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:02,950 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:08,048 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:08,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:13,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:14,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:54:14,458 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:54:18,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:18,714 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:23,275 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:54:24,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:28,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:29,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:29,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:54:29,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:54:34,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:38,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:40,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:44,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:54:44,456 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:54:45,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:46,342 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:50,618 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:53,276 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:54:54,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:54:56,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:54:59,458 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:54:59,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:54:59,461 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:55:01,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:04,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:07,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:12,974 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:14,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:55:14,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:55:14,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:18,686 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:23,276 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:55:24,277 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:24,602 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:29,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:29,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:55:29,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:55:29,459 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:55:32,666 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:34,527 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:40,521 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:42,742 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:44,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:55:44,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:55:45,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:50,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:51,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:53,277 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:55:57,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:55:57,848 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:55:59,457 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:55:59,457 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:56:02,684 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:07,915 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:08,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:13,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:14,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:56:14,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:56:14,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:56:18,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:19,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:23,278 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:56:24,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:26,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:29,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:56:29,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:56:29,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:29,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:56:34,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:56:34,596 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:56:34,597 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:56:34,598 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:56:35,170 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:56:35,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:36,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:40,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:44,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:56:44,942 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:56:46,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:46,240 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:51,167 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:53,278 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:56:54,306 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:56:56,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:56:59,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:56:59,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:56:59,945 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:57:02,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:04,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:07,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:13,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:14,431 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:14,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:57:14,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:57:19,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:22,480 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:23,279 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:57:24,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:29,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:29,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:57:29,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:57:32,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:35,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:40,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:42,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:44,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:57:44,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:57:45,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:50,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:52,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:57:53,279 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:57:56,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:57:59,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:57:59,943 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:58:00,725 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:02,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:07,159 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:10,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:12,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:15,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:58:15,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:58:15,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:58:18,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:18,830 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:23,280 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:58:24,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:26,890 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:29,655 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:30,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:58:30,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:58:34,998 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:36,953 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:40,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:45,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:45,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:58:45,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:58:47,015 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:58:50,955 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:58:53,291 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:58:56,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:00,102 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:00,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:59:00,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:59:00,773 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:59:02,030 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:07,032 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:08,152 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:12,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:15,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 08:59:15,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:59:15,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:59:17,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:18,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:22,997 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:23,282 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:59:28,281 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:28,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:30,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:59:30,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:59:33,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:38,344 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:39,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:44,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:45,006 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 08:59:45,007 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 08:59:45,007 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 08:59:45,010 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 08:59:45,391 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 08:59:45,773 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 08:59:45,774 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 08:59:46,395 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 08:59:49,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:53,283 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 08:59:55,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 08:59:56,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:00,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:02,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:00:02,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:00:02,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:00:04,553 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:06,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:11,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:14,624 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:16,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:17,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:00:17,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:00:17,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:00:22,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:23,284 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:00:24,695 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:27,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:32,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:00:32,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:00:32,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:00:32,293 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:34,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:37,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:43,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:44,876 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:47,130 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:00:47,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:00:47,131 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:00:48,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:52,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:00:53,284 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:00:54,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:00:59,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:02,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:01:02,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:01:02,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:01:03,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:04,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:09,387 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:13,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:14,616 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:17,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:01:17,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:01:17,135 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:01:19,930 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:21,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:23,285 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:01:25,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:30,361 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:30,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:32,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:01:32,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:01:32,132 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:01:36,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:40,433 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:41,488 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:46,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:47,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:01:47,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:01:47,134 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:01:48,614 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:01:52,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:53,286 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:01:57,176 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:01:58,675 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:02,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:02:02,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:02:02,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:02:02,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:07,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:08,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:12,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:16,908 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:17,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:02:17,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:02:17,133 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:02:18,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:23,287 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:02:24,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:26,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:29,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:32,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:02:32,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:02:32,417 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:02:35,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:35,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:40,463 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:45,091 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:46,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:47,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:02:47,416 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:02:51,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:02:51,246 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:02:51,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:02:51,249 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:02:52,139 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:02:52,250 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:02:53,144 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:02:53,287 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:02:57,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:02,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:03:02,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:02,416 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:03:03,392 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:07,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:10,454 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:13,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:17,420 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:03:17,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:03:17,424 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:03:18,631 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:20,528 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:23,287 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:03:24,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:28,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:29,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:32,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:03:32,421 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:03:34,630 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:38,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:39,664 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:44,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:46,710 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:03:47,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:03:47,422 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:03:50,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:53,290 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:03:56,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:03:56,773 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:01,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:03,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:04:03,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:04:03,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:04:04,833 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:06,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:11,179 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:14,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:17,175 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:18,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:04:18,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:04:22,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:23,290 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:04:24,954 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:28,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:33,009 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:33,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:04:33,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:04:33,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:04:33,313 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:38,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:43,067 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:43,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:48,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:04:48,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:04:48,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:04:49,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:04:53,138 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:04:53,290 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:04:55,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:00,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:01,236 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:03,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:05:03,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:05:06,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:11,307 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:11,937 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:16,938 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:18,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:05:18,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:05:19,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:22,332 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:23,291 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:05:28,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:29,432 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:33,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:05:33,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:05:33,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:38,771 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:39,498 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:43,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:47,552 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:48,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:05:48,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:05:49,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:53,294 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:05:54,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:05:54,956 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:05:54,957 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:05:54,958 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:05:54,960 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:05:55,599 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:05:56,603 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:57,607 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:05:59,962 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:03,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:06:03,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:06:04,654 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:05,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:10,221 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:14,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:15,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:18,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:06:18,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:06:21,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:23,293 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:06:24,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:26,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:31,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:33,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:06:33,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:06:34,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:36,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:41,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:42,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:46,717 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:48,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:06:48,108 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:06:52,177 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:06:53,008 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:06:53,293 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:06:57,294 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:01,061 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:02,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:03,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:07:03,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:07:08,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:11,124 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:14,206 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:18,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:07:18,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:07:19,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:21,187 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:23,294 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:07:25,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:29,238 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:30,519 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:33,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:07:33,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:07:36,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:39,304 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:41,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:46,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:47,363 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:48,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:07:48,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:07:52,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:07:53,296 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:07:57,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:07:57,441 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:02,442 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:03,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:08:03,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:08:08,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:13,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:17,633 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:08:18,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:08:18,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:08:18,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:23,295 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:08:24,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:27,700 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:08:29,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:34,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:34,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:08:34,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:08:34,452 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:08:37,769 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:08:39,657 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:44,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:47,828 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:08:49,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:08:49,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:08:49,450 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:08:49,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:08:53,297 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:08:54,869 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:08:55,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:00,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:04,449 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:09:04,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:09:04,451 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:09:04,937 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:05,612 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:10,614 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:12,825 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:09:12,826 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:09:12,826 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:09:12,828 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:09:13,002 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:09:13,002 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:15,014 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:15,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:20,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:09:20,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:09:20,737 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:09:20,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:23,077 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:23,297 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:09:26,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:31,945 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:33,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:35,734 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:09:35,735 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:09:37,919 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:41,230 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:43,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:48,526 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:50,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:09:50,736 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:09:51,283 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:09:53,298 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:09:54,298 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:09:59,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:01,347 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:04,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:07,225 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:10:07,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:10:07,227 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:10:09,406 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:10,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:15,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:19,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:21,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:22,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:10:22,225 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:10:23,298 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:10:27,295 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:27,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:32,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:37,224 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:10:37,225 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:10:37,382 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:37,600 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:42,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:45,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:47,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:52,870 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:10:53,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:10:53,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:10:53,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:10:53,336 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:10:55,705 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:10:58,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:03,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:05,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:08,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:11:08,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:11:08,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:11:09,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:13,832 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:14,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:19,761 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:23,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:11:23,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:11:23,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:11:23,325 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:11:23,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:25,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:30,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:31,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:35,659 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:38,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:11:38,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:11:38,770 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:11:40,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:42,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:45,890 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:51,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:51,122 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:11:53,300 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:11:53,767 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:11:53,768 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:11:56,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:11:59,183 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:01,992 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:06,993 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:08,768 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:12:08,768 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:12:09,258 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:12,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:16,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:12:16,918 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:12:16,919 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:12:16,921 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:12:17,323 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:12:17,324 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:18,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:19,336 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:23,302 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:12:24,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:25,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:12:25,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:12:25,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:12:27,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:29,443 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:34,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:39,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:40,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:12:40,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:12:41,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:44,693 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:49,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:51,536 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:12:53,302 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:12:55,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:12:55,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:12:55,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:12:59,601 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:00,987 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:05,988 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:09,660 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:10,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:13:10,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:13:11,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:16,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:19,722 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:22,215 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:23,305 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:13:25,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:13:25,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:13:25,148 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:13:27,273 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:27,777 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:32,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:35,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:38,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:40,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:13:40,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:13:40,146 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:13:43,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:45,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:48,601 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:53,305 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:13:53,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:13:55,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:13:55,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:13:55,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:13:55,969 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:13:59,323 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:04,036 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:04,687 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:09,689 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:10,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:14:10,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:14:10,889 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:14:14,110 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:15,029 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:20,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:23,305 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:14:24,176 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:25,306 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:25,887 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:14:25,888 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:14:30,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:32,231 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:35,695 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:40,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:40,888 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:14:40,935 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:14:40,936 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:14:42,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:46,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:49,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:14:51,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:53,305 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:14:56,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:14:56,645 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:14:56,646 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:14:56,859 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:14:59,439 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:02,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:07,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:08,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:11,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:15:11,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:15:13,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:17,583 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:19,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:23,307 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:15:24,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:25,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:15:25,559 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:15:25,559 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:15:25,561 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:15:25,646 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:15:26,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:15:26,645 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:15:27,659 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:29,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:35,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:35,718 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:40,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:42,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:15:42,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:15:42,262 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:15:45,782 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:46,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:51,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:15:53,307 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:15:53,838 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:15:57,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:15:57,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:15:57,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:15:57,459 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:02,460 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:03,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:07,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:12,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:16:12,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:16:12,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:16:13,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:13,977 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:18,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:22,033 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:23,308 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:16:24,309 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:27,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:16:27,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:16:27,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:16:29,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:32,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:34,446 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:40,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:40,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:42,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:16:42,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:16:42,268 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:16:45,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:50,261 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:16:50,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:53,308 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:16:56,310 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:16:57,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:16:57,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:16:57,264 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:17:00,333 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:01,967 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:06,969 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:07,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:12,141 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:12,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:17:12,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:17:12,262 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:17:17,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:17,641 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:22,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:23,310 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:17:27,260 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:17:27,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:17:27,261 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:17:27,497 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:27,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:32,498 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:35,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:37,650 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:42,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:17:42,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:17:42,592 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:17:42,840 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:45,820 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:48,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:53,310 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:17:53,887 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:17:54,311 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:17:57,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:17:57,593 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:17:59,760 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:03,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:05,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:11,529 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:12,013 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:12,600 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:18:12,619 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:18:16,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:21,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:22,090 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:23,311 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:18:27,312 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:27,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:18:27,777 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:18:27,779 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:18:28,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:18:28,996 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:18:28,996 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:18:28,997 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:18:29,175 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:18:30,180 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:32,999 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:38,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:40,247 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:42,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:18:42,776 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:18:42,777 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:18:43,979 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:48,300 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:49,000 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:53,314 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:18:54,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:18:57,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:18:57,776 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:18:58,365 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:18:59,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:05,925 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:06,416 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:10,926 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:13,545 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:19:13,546 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:19:16,476 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:16,739 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:21,740 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:23,313 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:19:26,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:27,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:28,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:19:28,546 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:19:32,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:34,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:37,909 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:43,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:19:43,547 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:19:43,794 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:44,650 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:48,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:52,707 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:19:53,313 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:19:54,314 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:19:59,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:19:59,143 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:19:59,144 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:19:59,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:02,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:05,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:10,827 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:11,132 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:14,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:20:14,142 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:20:16,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:19,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:21,450 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:23,314 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:20:27,315 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:27,930 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:29,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:20:29,143 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:20:32,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:37,998 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:38,286 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:43,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:45,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:20:45,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:20:45,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:20:48,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:49,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:53,316 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:20:54,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:20:56,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:20:59,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:00,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:21:00,164 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:21:05,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:06,192 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:10,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:15,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:21:15,164 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:21:15,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:16,255 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:20,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:23,316 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:21:24,305 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:25,626 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:30,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:21:30,165 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:21:31,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:33,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:21:33,046 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:21:33,046 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:21:33,048 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:21:33,372 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:21:34,379 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:37,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:42,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:44,450 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:45,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:21:45,164 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:21:47,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:52,507 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:21:52,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:21:53,316 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:21:58,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:00,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:22:00,166 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:22:02,579 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:03,503 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:08,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:10,646 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:14,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:15,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:22:15,165 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:22:20,094 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:20,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:23,317 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:22:25,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:30,164 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:22:30,165 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:22:30,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:30,789 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:35,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:40,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:40,850 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:45,536 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:46,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:22:46,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:22:46,552 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:22:48,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:22:50,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:53,317 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:22:56,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:22:58,992 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:01,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:23:01,550 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:23:01,551 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:23:01,774 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:06,065 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:06,791 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:11,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:16,133 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:16,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:23:16,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:23:16,551 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:23:16,813 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:21,814 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:23,318 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:23:26,209 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:27,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:31,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:23:31,977 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:23:31,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:23:32,231 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:34,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:38,161 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:42,350 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:44,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:46,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:23:46,976 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:23:49,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:52,407 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:23:53,319 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:23:54,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:23:59,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:01,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:24:01,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:24:01,977 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:24:02,506 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:05,146 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:10,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:10,577 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:15,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:17,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:24:17,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:24:17,401 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:24:20,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:20,647 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:23,319 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:24:26,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:28,704 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:31,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:32,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:24:32,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:24:36,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:37,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:24:37,146 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:24:37,147 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:24:37,148 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:24:37,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:24:37,772 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:38,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:42,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:47,223 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:47,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:24:47,400 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:24:48,835 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:52,590 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:24:53,320 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:24:56,886 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:24:58,113 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:03,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:03,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:25:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:25:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:25:06,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:08,701 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:13,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:17,025 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:18,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:25:18,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:25:18,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:25:18,766 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:23,321 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:25:24,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:25,085 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:29,020 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:33,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:25:33,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:25:33,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:25:34,779 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:35,153 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:39,780 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:43,208 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:45,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:48,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:25:48,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:25:48,558 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:25:50,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:25:53,271 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:25:53,321 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:25:56,322 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:01,324 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:03,330 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:03,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:26:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:26:03,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:26:06,769 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:11,770 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:16,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:17,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:18,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:26:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:26:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:26:22,822 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:23,324 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:26:26,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:28,464 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:33,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:26:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:26:33,555 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:26:33,729 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:34,619 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:38,730 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:42,676 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:44,169 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:26:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:26:48,557 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:26:49,796 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:26:52,749 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:26:53,324 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:26:55,325 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:00,327 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:02,811 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:27:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:27:03,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:27:05,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:10,874 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:11,034 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:16,035 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:18,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:27:18,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:27:18,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:27:20,933 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:21,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:23,325 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:27:27,326 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:28,991 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:32,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:33,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:27:33,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:27:33,557 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:27:37,995 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:39,064 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:42,996 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:45,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:27:45,977 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:27:45,978 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:27:45,980 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:27:46,116 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:27:47,121 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:48,126 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:48,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:27:48,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:27:48,556 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:27:48,764 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:53,327 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:27:54,329 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:27:57,184 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:27:59,881 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:03,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:28:03,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:28:03,956 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:28:05,174 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:07,254 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:10,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:15,304 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:15,317 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:18,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:28:18,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:28:18,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:28:21,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:23,326 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:28:25,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:26,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:31,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:33,441 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:33,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:28:33,953 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:28:33,954 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:28:37,163 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:42,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:43,507 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:47,249 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:49,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:28:49,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:28:49,107 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:28:50,557 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:28:52,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:28:53,329 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:28:57,330 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:00,622 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:03,155 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:04,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:29:04,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:29:04,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:29:08,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:09,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:14,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:18,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:19,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:29:19,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:29:19,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:29:19,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:23,330 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:29:25,331 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:28,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:31,321 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:35,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:29:35,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:29:35,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:29:36,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:36,844 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:41,437 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:46,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:46,904 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:50,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:29:50,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:29:50,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:29:52,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:29:53,331 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:29:56,978 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:29:57,961 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:02,963 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:05,031 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:05,122 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:30:05,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:30:05,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:30:08,303 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:13,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:15,084 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:18,704 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:20,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:30:20,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:30:20,126 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:30:23,331 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:30:24,427 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:25,146 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:29,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:35,043 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:35,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:30:35,124 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:30:35,124 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:30:35,302 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:40,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:43,362 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:45,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:50,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:30:51,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:30:51,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:30:51,117 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:30:52,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:30:52,255 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:30:52,256 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:30:52,258 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:30:52,422 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:30:53,336 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:30:53,425 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:54,429 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:30:56,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:01,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:03,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:06,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:31:06,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:31:06,115 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:31:07,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:13,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:13,549 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:18,106 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:21,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:31:21,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:31:21,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:31:23,343 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:31:23,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:23,617 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:25,231 ERROR gpu :60204 [interfaces.py:aggregate():161] Failed to serialize metric: The operating system has blocked the request. 2023-11-19 09:31:28,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:33,687 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:33,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:36,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:31:36,115 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:31:39,296 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:41,738 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:45,138 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:50,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:31:51,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:31:51,115 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:31:51,798 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:31:53,343 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:31:56,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:00,851 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:02,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:07,070 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:07,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:32:07,121 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:32:07,123 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:32:08,899 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:12,267 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:17,269 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:18,951 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:22,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:32:22,120 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:32:22,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:23,345 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:32:27,975 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:29,019 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:32,976 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:37,073 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:37,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:32:37,121 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:32:38,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:43,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:47,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:48,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:53,346 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:32:53,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:32:53,525 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:32:53,527 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:32:53,743 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:32:55,222 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:32:59,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:04,719 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:05,276 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:08,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:33:08,523 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:33:10,685 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:13,322 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:15,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:20,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:23,346 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:33:23,386 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:23,523 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:33:23,523 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:33:26,745 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:32,195 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:33,453 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:37,196 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:39,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:33:39,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:33:39,495 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:33:41,510 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:42,829 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:47,831 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:51,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:52,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:33:53,347 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:33:54,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:33:54,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:33:54,491 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:33:55,591 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:33:58,677 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:00,666 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:34:00,668 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:34:00,668 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:34:00,670 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:34:01,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:34:01,652 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:02,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:04,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:09,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:34:09,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:34:09,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:34:09,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:11,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:14,821 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:20,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:21,802 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:23,347 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:34:24,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:34:24,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:34:24,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:34:25,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:29,866 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:30,952 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:35,954 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:39,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:34:39,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:34:39,492 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:34:39,936 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:41,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:46,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:50,006 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:52,349 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:34:53,348 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:34:55,101 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:34:55,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:34:55,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:34:57,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:34:58,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:03,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:07,113 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:08,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:10,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:35:10,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:35:10,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:35:14,319 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:17,188 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:19,385 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:23,350 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:35:24,513 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:25,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:35:25,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:35:25,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:35:25,243 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:30,271 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:35,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:36,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:40,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:35:40,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:35:40,103 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:35:41,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:45,387 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:47,287 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:52,288 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:35:53,351 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:35:55,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:35:55,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:35:55,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:35:55,544 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:35:57,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:02,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:05,615 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:07,871 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:10,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:36:10,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:36:10,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:36:13,262 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:15,865 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:18,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:23,351 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:36:23,352 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:23,921 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:25,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:36:25,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:36:25,104 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:36:29,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:33,988 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:35,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:40,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:40,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:36:40,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:36:40,106 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:36:44,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:45,517 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:50,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:36:52,140 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:36:53,352 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:36:55,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:36:55,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:36:55,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:36:56,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:02,004 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:02,206 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:07,005 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:09,671 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:37:09,672 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:37:09,672 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:37:09,676 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:37:10,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:37:10,104 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:37:10,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:37:10,256 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:37:11,260 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:12,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:17,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:19,308 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:23,166 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:23,352 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:37:25,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:37:25,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:37:25,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:37:28,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:29,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:33,637 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:38,772 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:39,430 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:40,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:37:40,105 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:37:40,105 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:37:44,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:49,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:37:49,694 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:53,353 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:37:55,354 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:37:55,490 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:37:55,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:37:55,493 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:37:57,629 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:00,700 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:06,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:07,699 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:10,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:38:10,490 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:38:11,727 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:16,835 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:17,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:21,836 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:23,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:38:25,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:38:25,490 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:38:25,809 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:27,646 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:32,648 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:35,878 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:38,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:41,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:38:41,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:38:41,907 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:38:44,045 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:45,935 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:49,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:53,354 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:38:53,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:38:54,355 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:38:56,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:38:56,905 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:39:00,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:04,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:05,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:10,193 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:11,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:39:11,905 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:39:12,096 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:16,052 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:21,220 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:22,160 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:23,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:39:26,356 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:27,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:39:27,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:39:27,473 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:39:30,212 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:31,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:36,806 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:40,296 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:42,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:42,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:39:42,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:39:42,516 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:39:47,723 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:50,368 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:52,738 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:39:53,355 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:39:57,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:39:57,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:39:57,473 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:39:58,424 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:39:58,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:03,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:08,682 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:11,503 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:12,472 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:40:12,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:40:12,474 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:40:13,731 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:18,733 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:20,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:40:20,937 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:40:20,938 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:40:20,939 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:40:21,573 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:40:21,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:23,356 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:40:23,623 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:24,357 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:28,907 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:40:28,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:40:28,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:40:30,037 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:31,919 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:35,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:40,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:42,018 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:43,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:40:43,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:40:43,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:40:45,157 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:50,075 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:40:50,658 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:53,356 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:40:56,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:40:58,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:40:58,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:41:00,147 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:02,123 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:07,125 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:10,204 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:12,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:13,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:41:13,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:41:17,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:18,264 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:22,949 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:23,357 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:41:28,316 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:28,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:41:28,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:28,908 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:41:34,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:36,375 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:40,042 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:43,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:41:43,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:41:45,139 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:46,534 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:50,140 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:53,358 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:41:54,586 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:41:55,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:41:58,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:41:58,909 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:42:01,076 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:03,646 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:06,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:11,440 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:13,706 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:13,908 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:42:13,910 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:42:17,165 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:21,750 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:23,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:23,360 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:42:28,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:29,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:42:29,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:42:29,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:42:31,826 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:33,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:38,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:41,910 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:44,012 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:44,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:42:44,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:42:44,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:42:49,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:51,968 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:42:53,359 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:42:54,360 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:42:59,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:42:59,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:42:59,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:43:00,254 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:02,028 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:05,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:10,078 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:10,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:14,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:43:14,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:43:14,062 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:43:16,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:20,135 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:21,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:23,361 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:43:26,479 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:26,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:43:26,482 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:43:26,482 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:43:26,485 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:43:27,194 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:43:27,195 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:28,199 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:29,060 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:43:29,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:43:29,061 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:43:32,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:37,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:38,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:42,456 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:44,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:43:44,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:43:44,063 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:43:46,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:47,673 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:52,675 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:53,361 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:43:56,396 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:43:58,602 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:43:59,061 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:43:59,062 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:43:59,062 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:44:04,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:09,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:14,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:15,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:44:15,285 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:44:15,287 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:44:18,527 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:44:19,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:23,363 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:44:25,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:28,580 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:44:30,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:44:30,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:44:30,239 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:44:30,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:35,435 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:38,653 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:44:40,485 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:45,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:44:45,240 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:44:45,240 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:44:46,451 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:48,732 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:44:51,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:44:53,364 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:44:56,795 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:44:56,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:00,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:45:00,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:45:00,239 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:45:02,439 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:05,856 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:07,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:12,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:13,917 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:15,237 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:45:15,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:45:15,239 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:45:18,489 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:23,364 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:45:23,980 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:24,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:29,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:30,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:45:30,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:45:30,243 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:45:34,040 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:34,959 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:39,960 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:42,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:45,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:45:45,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:45,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:45:45,241 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:45:50,487 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:45:52,156 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:45:53,364 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:45:56,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:00,238 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:46:00,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:46:00,241 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:46:01,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:02,216 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:06,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:10,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:12,017 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:15,239 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:46:15,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:46:15,242 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:46:17,415 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:20,348 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:23,182 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:23,365 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:46:28,403 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:29,099 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:30,245 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:46:30,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:46:30,246 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:46:34,508 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:38,468 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:40,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:45,131 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:45,247 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:46:45,248 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:46:45,249 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:46:46,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:46:46,257 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:46:46,257 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:46:46,260 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:46:46,528 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:46:47,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:48,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:46:50,261 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:53,367 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:46:56,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:46:56,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:00,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:47:00,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:47:01,462 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:06,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:07,226 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:12,228 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:14,741 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:15,246 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:47:15,247 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:47:17,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:23,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:23,367 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:47:24,806 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:28,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:30,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:47:30,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:47:30,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:47:33,920 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:34,871 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:38,922 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:42,931 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:44,142 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:45,333 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:47:45,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:47:49,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:47:52,093 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:47:53,367 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:47:55,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:00,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:48:00,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:48:00,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:02,169 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:05,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:10,222 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:11,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:15,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:48:15,334 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:48:17,491 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:20,472 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:22,546 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:23,368 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:48:28,502 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:28,533 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:30,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:48:30,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:48:30,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:48:33,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:38,593 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:39,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:44,301 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:45,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:48:45,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:48:45,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:48:48,672 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:48:49,471 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:53,382 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:48:55,383 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:48:56,771 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:00,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:49:00,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:49:00,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:05,757 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:06,834 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:10,758 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:15,334 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:49:15,335 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:49:16,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:16,898 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:21,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:23,370 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:49:26,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:26,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:30,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:49:30,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:49:30,339 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:49:32,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:35,012 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:38,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:44,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:45,071 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:45,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:49:45,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:49:49,572 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:49:51,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 09:49:51,418 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 09:49:51,418 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 09:49:51,425 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 09:49:52,118 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 09:49:53,124 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:53,370 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:49:54,128 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:55,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 09:49:55,371 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:00,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:50:00,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:50:00,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:05,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:10,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:15,335 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:50:15,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:50:15,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:20,577 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:23,371 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:50:26,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:30,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:50:30,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:50:31,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:36,537 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:41,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:45,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:50:45,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:50:45,336 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:50:47,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:52,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:50:53,371 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:50:58,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:00,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:51:00,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:51:00,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:51:03,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:08,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:13,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:15,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:51:15,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:51:15,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:51:18,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:23,372 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:51:24,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:29,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:30,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:51:30,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:51:34,539 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:39,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:44,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:45,336 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:51:45,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:51:50,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:51:53,372 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:51:56,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:00,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:52:00,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:52:01,547 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:06,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:11,548 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:15,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:52:15,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:52:17,530 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:22,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:23,372 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:52:28,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:30,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:52:30,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:52:30,338 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:52:33,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:38,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:43,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:45,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:52:45,337 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:52:48,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:53,373 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:52:54,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:52:59,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:00,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:53:00,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:53:00,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:53:04,596 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:09,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:14,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:15,337 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:53:15,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:53:15,341 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:53:20,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:23,373 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:53:26,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:30,338 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:53:30,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:53:30,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:53:31,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:36,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:41,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:45,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:53:45,358 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:53:45,364 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:53:46,587 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:51,588 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:53:53,374 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:53:57,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:00,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:54:00,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:54:00,347 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:54:02,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:07,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:12,570 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:15,339 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:54:15,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:54:15,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:54:17,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:22,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:23,375 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:54:28,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:30,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:54:30,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:54:30,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:54:33,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:38,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:43,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:45,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:54:45,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:54:45,342 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:54:48,535 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:53,375 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:54:54,375 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:54:59,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:00,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:55:00,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:55:00,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:55:04,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:09,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:14,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:15,340 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:55:15,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:55:15,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:55:20,532 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:23,375 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:55:26,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:30,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:55:30,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:55:30,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:55:31,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:36,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:41,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:45,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:55:45,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:55:45,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:55:47,515 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:52,516 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:55:53,376 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:55:58,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:00,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:56:00,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:56:00,343 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:56:03,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:08,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:13,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:15,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:56:15,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:56:15,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:56:18,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:23,377 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:56:24,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:29,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:30,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:56:30,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:56:30,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:56:34,505 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:39,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:44,506 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:45,341 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:56:45,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:56:45,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:56:49,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:56:53,377 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:56:55,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:00,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:57:00,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:57:00,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:57:00,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:05,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:10,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:15,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:57:15,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:57:15,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:57:16,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:21,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:23,378 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:57:27,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:30,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:57:30,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:57:30,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:57:32,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:37,594 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:42,595 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:45,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:57:45,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:57:45,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:57:48,553 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:53,378 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:57:54,379 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:57:59,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:00,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:58:00,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:58:00,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:58:04,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:09,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:14,560 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:15,342 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:58:15,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:58:15,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:58:20,531 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:23,379 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:58:26,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:30,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:58:30,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:58:30,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:58:31,591 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:36,592 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:41,593 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:45,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:58:45,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:58:45,344 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:58:47,582 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:52,583 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:58:53,380 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:58:58,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:00,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:59:00,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:59:00,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:59:03,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:08,574 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:13,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:15,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:59:15,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:59:15,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:59:18,586 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:23,380 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:59:24,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:29,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:30,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:59:30,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:59:30,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:59:34,554 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:39,555 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:44,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:45,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 09:59:45,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 09:59:45,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 09:59:50,542 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 09:59:53,385 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 09:59:56,386 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:00,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:00:00,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:00:00,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:00:01,492 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:06,493 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:11,494 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:15,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:00:15,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:00:15,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:00:16,598 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:21,599 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:23,386 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:00:27,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:30,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:00:30,346 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:00:30,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:00:32,510 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:37,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:42,512 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:45,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:00:45,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:00:45,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:00:47,575 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:52,576 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:00:53,386 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:00:58,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:00,263 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 10:01:00,265 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 10:01:00,266 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 10:01:00,268 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 10:01:00,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:01:00,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:01:00,345 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:01:00,753 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 10:01:01,756 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:01:02,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:01:03,640 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:08,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:13,641 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:15,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:01:15,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:01:15,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:01:19,609 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:23,387 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:01:25,388 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:30,344 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:01:30,345 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:01:30,346 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:01:30,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:35,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:40,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:45,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:01:45,972 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:45,981 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:01:46,289 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:01:46,289 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:01:51,528 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:01:53,388 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:01:57,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:00,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:02:01,499 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:02:01,499 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:02:02,946 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:07,947 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:12,948 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:15,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:02:15,874 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:02:15,874 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:02:18,520 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:23,397 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:02:24,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:29,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:30,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:02:30,798 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:02:30,799 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:02:34,628 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:36,269 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:02:37,274 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:02:40,256 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:45,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:50,258 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:53,389 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:02:55,389 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:02:56,205 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:02:56,300 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: stop_status 2023-11-19 10:02:56,300 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: stop_status 2023-11-19 10:02:58,390 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:03:00,566 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:05,567 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:10,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:15,568 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:20,468 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:03:20,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:23,389 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:03:26,390 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:31,391 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:36,392 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:41,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:46,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:51,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:03:53,391 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:03:57,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:02,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:07,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:10,351 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:10,359 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:12,416 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:12,775 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:04:16,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:17,417 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:21,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:22,418 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:23,391 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:04:28,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:31,363 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:33,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:36,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:38,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:44,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:45,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:49,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:04:50,366 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:04:53,392 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:04:54,393 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:00,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:00,367 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:05,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:05,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:11,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:15,369 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:16,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:18,053 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:05:20,372 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:21,421 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:23,392 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:05:25,374 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:27,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:32,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:35,054 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:37,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:38,172 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:05:40,056 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:43,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:46,088 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:48,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:50,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:05:51,089 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:53,393 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:05:54,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:05:57,063 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:57,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:05:59,678 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:00,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:02,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:04,679 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:08,350 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:09,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:12,413 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:15,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:18,579 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:20,944 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:21,502 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:23,393 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:06:23,580 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:26,394 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:31,571 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:32,364 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:32,778 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:37,365 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:37,830 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:41,632 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:43,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:47,790 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:48,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:52,834 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:06:53,243 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:06:53,394 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:06:53,712 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:06:59,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:02,480 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:03,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:05,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:07,481 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:10,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:13,889 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:15,272 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:18,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:20,274 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:23,259 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:23,395 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:07:23,959 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:25,396 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:31,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:33,018 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:34,052 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:36,255 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:38,019 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:41,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:44,151 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:46,604 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:47,305 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:51,606 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:07:52,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:07:53,395 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:07:54,217 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:07:57,064 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:02,066 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:02,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:04,309 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:07,423 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:07,918 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:13,024 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:14,384 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:17,748 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:18,025 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:22,861 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:23,026 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:23,396 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:08:24,462 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:28,065 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:32,564 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:33,067 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:36,559 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:37,565 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:38,923 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:44,855 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:46,642 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:47,578 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:49,856 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:53,399 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:08:53,797 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:08:55,108 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:08:55,112 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 10:08:55,116 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 10:08:55,116 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 10:08:55,119 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 10:08:55,751 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 10:08:55,752 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:08:56,757 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:00,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:02,800 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:05,597 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:07,857 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:08,728 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:10,942 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:16,819 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:17,608 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:17,957 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:21,820 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:22,610 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:23,399 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:09:27,469 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:28,027 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:32,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:34,102 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:37,735 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:38,119 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:39,107 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:43,556 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:48,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:49,434 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:09:50,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:09:53,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:09:53,399 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:09:55,400 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:00,332 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:00,897 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:03,538 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:05,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:09,058 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:10,899 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:16,398 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:18,028 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:18,526 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:21,399 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:23,031 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:23,401 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:10:26,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:28,606 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:31,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:33,216 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:36,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:39,072 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:40,691 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:42,448 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:48,046 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:48,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:48,747 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:53,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:10:53,194 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:10:53,400 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:10:57,822 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:10:58,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:02,917 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:03,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:08,702 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:09,120 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:09,903 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:11:14,557 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:17,877 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:19,983 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:11:20,203 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:22,878 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:23,401 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:11:25,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:30,086 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:11:31,281 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:32,941 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:36,283 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:38,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:40,158 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:11:41,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:46,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:47,720 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:50,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:11:51,866 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:11:52,721 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:11:53,401 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:11:57,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:00,312 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:03,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:03,368 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:08,150 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:08,370 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:10,400 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:13,558 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:18,549 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:18,559 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:20,465 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:23,402 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:12:23,550 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:24,403 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:28,898 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 10:12:28,899 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 10:12:28,900 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 10:12:28,901 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 10:12:29,532 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 10:12:29,904 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:30,537 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:31,541 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:32,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:34,905 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:37,662 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:40,191 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:40,651 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:45,569 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:48,241 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:50,901 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:12:51,422 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:12:53,242 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:12:53,402 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:12:57,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:00,965 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:03,077 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:03,290 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:08,133 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:08,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:10,046 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:14,078 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:18,911 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:19,079 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:20,137 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:23,403 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:13:23,912 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:24,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:29,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:32,224 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:34,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:35,114 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:40,049 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:40,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:42,289 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:45,651 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:49,145 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:50,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:13:52,374 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:13:53,403 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:13:54,148 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:13:56,404 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:01,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:04,500 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:04,533 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:07,001 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:10,571 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:12,914 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:14,576 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:18,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:21,047 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:23,404 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:14:24,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:24,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:26,048 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:29,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:33,885 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:34,762 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:35,081 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:38,886 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:40,082 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:44,853 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:45,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:48,708 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:50,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:14:53,405 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:14:54,395 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:14:54,939 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:14:55,775 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:01,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:04,291 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:05,049 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:07,473 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:09,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:12,475 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:15,132 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:17,665 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:18,785 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:22,667 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:23,406 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:15:24,160 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:24,223 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:27,746 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:33,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:33,807 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:36,325 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:38,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:39,103 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:44,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:46,398 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:49,188 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:49,384 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:53,411 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:15:54,257 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 10:15:54,259 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 10:15:54,260 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 10:15:54,262 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 10:15:54,473 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 10:15:54,867 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:15:55,264 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:15:55,477 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:15:56,483 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:00,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:03,808 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:05,980 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:06,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:08,809 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:10,981 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:16,013 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:16,657 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:19,649 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:21,014 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:23,423 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:16:25,051 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:26,436 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:26,720 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:31,654 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:34,022 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:36,784 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:37,297 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:39,023 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:42,299 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:46,855 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:16:47,562 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:49,343 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:52,563 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:53,412 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:16:54,726 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:16:57,958 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:16:58,970 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:03,656 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:04,652 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:07,044 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:08,889 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:09,653 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:13,891 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:17,259 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:19,278 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:19,573 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:23,420 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:17:24,759 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:24,973 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:27,331 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:30,279 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:34,540 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:35,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:37,411 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:39,541 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:40,744 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:46,485 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:46,669 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:50,115 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:51,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:17:53,415 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:17:55,292 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:17:56,590 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:17:57,534 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:03,280 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:05,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:06,658 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:18:08,452 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:10,252 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:13,454 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:16,726 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:18:18,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:20,377 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:23,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:18:24,433 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:25,511 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:26,817 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:18:30,376 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:35,361 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:35,378 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:38,907 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:18:40,362 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:40,832 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:46,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:48,968 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:18:50,152 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:51,518 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:53,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:18:55,428 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:18:56,910 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:18:59,056 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:02,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:05,884 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:07,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:11,148 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:11,168 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:12,185 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:18,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:19,186 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: partial_history 2023-11-19 10:19:19,192 DEBUG SenderThread:60204 [sender.py:send():380] send: history 2023-11-19 10:19:19,193 DEBUG SenderThread:60204 [sender.py:send_request():407] send_request: summary_record 2023-11-19 10:19:19,196 INFO SenderThread:60204 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-19 10:19:19,207 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/wandb-summary.json 2023-11-19 10:19:20,213 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:20,642 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:21,218 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:23,198 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:23,429 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:19:25,644 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:28,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:31,298 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:34,127 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:35,681 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:39,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:40,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:43,397 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:44,320 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:50,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:51,068 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:19:53,429 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:19:53,490 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:19:55,430 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:19:56,069 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:01,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:03,574 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:05,425 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:06,233 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:10,670 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:11,851 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:12,643 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:17,128 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:20,444 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:22,680 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:22,730 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:23,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:20:25,445 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:28,432 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:32,815 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:33,619 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:35,927 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:38,620 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:40,943 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:43,106 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:44,270 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:50,129 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:50,929 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:20:53,430 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:20:55,214 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:20:55,431 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:20:55,936 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:00,465 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:01,316 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:01,317 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:05,299 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:05,802 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:06,318 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:10,805 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:11,812 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:15,409 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:16,380 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:21,381 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:23,038 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:23,431 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:21:26,402 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:27,492 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:28,039 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:31,405 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:37,149 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:37,656 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:37,792 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:42,151 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:43,713 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:47,183 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:47,728 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:52,793 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:53,433 DEBUG SenderThread:60204 [sender.py:send():380] send: stats 2023-11-19 10:21:54,697 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive 2023-11-19 10:21:56,823 INFO Thread-12 :60204 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_132020-qg88991p/files/output.log 2023-11-19 10:21:58,192 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: status_report 2023-11-19 10:21:59,698 DEBUG HandlerThread:60204 [handler.py:handle_request():146] handle_request: keepalive