2023-11-22 10:00:21,669 INFO StreamThr :2219461 [internal.py:wandb_internal():86] W&B internal server running at pid: 2219461, started at: 2023-11-22 10:00:21.667636 2023-11-22 10:00:21,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status 2023-11-22 10:00:21,720 INFO WriterThread:2219461 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231122_100021-qrxr9o47/run-qrxr9o47.wandb 2023-11-22 10:00:21,722 DEBUG SenderThread:2219461 [sender.py:send():380] send: header 2023-11-22 10:00:21,724 DEBUG SenderThread:2219461 [sender.py:send():380] send: run 2023-11-22 10:00:22,191 INFO SenderThread:2219461 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files 2023-11-22 10:00:22,192 INFO SenderThread:2219461 [sender.py:_start_run_threads():1124] run started: qrxr9o47 with start time 1700640021.677551 2023-11-22 10:00:22,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: check_version 2023-11-22 10:00:22,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: check_version 2023-11-22 10:00:22,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: run_start 2023-11-22 10:00:22,288 DEBUG HandlerThread:2219461 [system_info.py:__init__():32] System info init 2023-11-22 10:00:22,289 DEBUG HandlerThread:2219461 [system_info.py:__init__():47] System info init done 2023-11-22 10:00:22,289 INFO HandlerThread:2219461 [system_monitor.py:start():194] Starting system monitor 2023-11-22 10:00:22,290 INFO SystemMonitor:2219461 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-11-22 10:00:22,290 INFO HandlerThread:2219461 [system_monitor.py:probe():214] Collecting system info 2023-11-22 10:00:22,291 INFO SystemMonitor:2219461 [interfaces.py:start():190] Started cpu monitoring 2023-11-22 10:00:22,293 INFO SystemMonitor:2219461 [interfaces.py:start():190] Started disk monitoring 2023-11-22 10:00:22,294 INFO SystemMonitor:2219461 [interfaces.py:start():190] Started gpu monitoring 2023-11-22 10:00:22,295 INFO SystemMonitor:2219461 [interfaces.py:start():190] Started memory monitoring 2023-11-22 10:00:22,296 INFO SystemMonitor:2219461 [interfaces.py:start():190] Started network monitoring 2023-11-22 10:00:22,301 DEBUG HandlerThread:2219461 [system_info.py:probe():196] Probing system 2023-11-22 10:00:22,390 DEBUG HandlerThread:2219461 [gitlib.py:_init_repo():56] git repository is invalid 2023-11-22 10:00:22,390 DEBUG HandlerThread:2219461 [system_info.py:probe():244] Probing system done 2023-11-22 10:00:22,390 DEBUG HandlerThread:2219461 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-22T08:00:22.301922', 'startedAt': '2023-11-22T08:00:21.614508', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--torch_compile', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub', '--push_to_hub_model_id=WhisperLargeFinnishV3', '--resume_from_checkpoint=./checkpoint-26000'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.58251190185547}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} 2023-11-22 10:00:22,391 INFO HandlerThread:2219461 [system_monitor.py:probe():224] Finished collecting system info 2023-11-22 10:00:22,391 INFO HandlerThread:2219461 [system_monitor.py:probe():227] Publishing system info 2023-11-22 10:00:22,391 DEBUG HandlerThread:2219461 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-11-22 10:00:22,396 DEBUG HandlerThread:2219461 [system_info.py:_save_pip():68] Saving pip packages done 2023-11-22 10:00:22,398 DEBUG HandlerThread:2219461 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-11-22 10:00:23,195 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/requirements.txt 2023-11-22 10:00:23,196 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/conda-environment.yaml 2023-11-22 10:00:27,603 DEBUG HandlerThread:2219461 [system_info.py:_save_conda():87] Saving conda packages done 2023-11-22 10:00:27,607 INFO HandlerThread:2219461 [system_monitor.py:probe():229] Finished publishing system info 2023-11-22 10:00:27,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:27,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 10:00:27,900 DEBUG SenderThread:2219461 [sender.py:send():380] send: files 2023-11-22 10:00:27,901 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now 2023-11-22 10:00:27,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:00:27,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:00:27,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:00:28,114 DEBUG SenderThread:2219461 [sender.py:send():380] send: telemetry 2023-11-22 10:00:28,114 DEBUG SenderThread:2219461 [sender.py:send():380] send: config 2023-11-22 10:00:28,115 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 10:00:28,115 DEBUG SenderThread:2219461 [sender.py:send():380] send: telemetry 2023-11-22 10:00:28,115 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 10:00:28,116 WARNING SenderThread:2219461 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) 2023-11-22 10:00:28,116 DEBUG SenderThread:2219461 [sender.py:send():380] send: telemetry 2023-11-22 10:00:28,218 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/conda-environment.yaml 2023-11-22 10:00:28,219 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:00:28,220 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-metadata.json 2023-11-22 10:00:28,565 INFO wandb-upload_0:2219461 [upload_job.py:push():131] Uploaded file /tmp/tmpqmlwx2tnwandb/rh9trgj9-wandb-metadata.json 2023-11-22 10:00:30,227 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:00:32,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:36,258 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:00:37,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:38,266 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:00:42,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:42,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:00:42,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:00:42,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:00:48,110 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:53,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:00:54,329 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/config.yaml 2023-11-22 10:00:57,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:00:57,920 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:00:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:00:59,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:04,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:09,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:12,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:01:12,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:01:12,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:01:14,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:19,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:22,297 DEBUG SystemMonitor:2219461 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-11-22 10:01:22,299 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:01:24,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:27,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:01:27,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:01:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:01:30,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:35,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:40,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:42,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:01:42,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:01:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:01:46,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:51,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:52,300 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:01:56,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:01:57,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:01:57,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:01:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:02:02,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:07,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:12,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:12,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:02:12,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:02:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:02:17,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:22,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:22,300 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:02:27,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:27,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:02:27,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:02:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:02:33,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:38,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:42,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:02:42,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:02:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:02:44,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:48,851 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:02:50,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:52,300 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:02:55,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:02:57,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:02:57,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:02:57,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:03:01,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:06,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:11,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:12,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:03:12,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:03:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:03:17,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:22,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:22,301 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:03:27,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:27,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:03:27,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:03:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:03:33,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:38,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:42,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:03:42,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:03:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:03:43,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:48,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:52,301 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:03:53,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:03:57,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:03:57,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:03:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:03:59,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:04,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:09,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:12,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:04:12,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:04:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:04:15,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:20,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:22,302 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:04:25,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:27,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:04:27,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:04:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:04:31,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:36,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:41,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:42,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:04:42,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:04:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:04:46,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:51,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:52,303 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:04:56,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:04:57,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:04:57,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:04:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:05:02,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:07,152 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:12,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:12,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:05:12,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:05:12,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:05:17,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:22,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:22,303 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:05:27,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:27,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:05:27,922 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:05:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:05:33,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:38,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:42,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:05:42,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:05:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:05:44,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:49,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:52,304 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:05:54,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:05:57,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:05:57,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:05:57,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:06:00,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:05,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:10,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:12,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:06:12,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:06:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:06:16,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:21,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:22,304 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:06:26,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:27,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:06:27,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:06:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:06:32,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:37,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:42,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:42,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:06:42,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:06:42,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:06:48,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:52,305 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:06:53,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:06:57,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:06:57,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:06:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:06:59,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:04,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:09,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:12,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:07:12,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:07:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:07:15,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:20,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:22,305 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:07:25,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:27,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:07:27,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:07:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:07:31,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:36,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:41,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:42,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:07:42,927 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:07:42,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:07:46,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:51,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:52,306 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:07:56,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:07:57,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:07:57,928 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:07:57,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:08:02,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:07,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:12,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:12,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:08:12,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:08:12,937 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:08:18,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:22,306 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:08:23,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:27,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:08:27,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:08:27,947 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:08:29,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:34,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:39,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:42,945 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:08:42,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:08:42,946 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:08:44,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:49,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:52,307 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:08:54,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:08:57,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:08:57,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:08:57,947 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:09:00,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:05,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:10,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:12,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:09:12,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:09:12,947 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:09:15,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:20,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:22,307 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:09:25,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:27,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:09:27,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:09:27,955 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:09:31,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:36,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:41,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:42,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:09:42,947 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:09:42,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:09:47,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:52,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:52,308 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:09:57,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:09:57,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:09:57,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:09:57,957 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:10:03,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:08,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:12,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:10:12,956 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:10:13,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:10:13,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:18,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:22,309 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:10:23,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:27,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:10:27,956 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:10:28,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:10:29,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:34,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:39,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:42,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:10:42,956 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:10:43,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:10:44,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:49,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:52,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:10:54,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:10:57,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:10:57,956 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:10:58,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:11:00,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:05,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:10,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:12,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:11:12,956 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:11:13,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:11:16,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:21,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:22,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:11:26,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:27,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:11:28,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:11:28,013 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:11:32,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:37,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:42,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:42,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:11:43,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:11:43,013 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:11:48,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:52,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:11:53,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:11:57,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:11:58,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:11:58,013 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:11:59,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:04,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:09,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:12,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:12:13,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:12:13,013 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:12:15,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:20,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:22,313 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:12:25,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:27,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:12:28,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:12:28,013 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:12:31,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:36,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:41,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:42,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:12:43,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:12:43,016 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:12:47,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:52,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:52,311 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:12:57,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:12:57,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:12:58,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:12:58,016 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:13:03,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:08,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:12,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:13:13,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:13:13,018 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:13:14,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:19,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:22,312 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:13:24,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:27,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:13:28,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:13:28,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:13:30,240 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:35,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:40,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:42,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:13:43,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:13:43,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:13:45,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:50,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:52,312 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:13:55,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:13:57,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:13:58,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:13:58,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:14:01,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:06,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:11,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:12,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:14:13,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:14:13,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:14:17,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:22,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:22,313 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:14:27,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:27,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:14:28,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:14:28,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:14:33,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:38,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:42,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:14:43,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:14:43,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:14:43,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:48,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:52,314 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:14:53,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:14:57,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:14:58,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:14:58,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:14:59,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:04,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:05,634 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:15:09,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:12,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:15:13,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:15:13,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:15:15,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:20,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:22,314 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:15:25,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:27,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:15:28,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:15:28,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:15:31,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:36,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:41,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:42,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:15:43,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:15:43,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:15:47,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:52,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:52,315 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:15:57,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:15:57,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:15:58,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:15:58,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:16:03,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:08,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:12,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:16:13,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:16:13,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:16:14,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:19,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:22,316 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:16:24,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:27,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:16:28,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:16:28,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:16:30,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:35,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:40,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:42,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:16:43,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:16:43,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:16:46,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:51,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:52,316 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:16:56,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:16:57,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:16:58,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:16:58,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:17:02,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:07,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:12,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:12,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:17:13,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:17:13,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:17:18,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:22,317 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:17:23,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:27,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:17:28,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:17:28,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:17:29,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:34,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:39,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:42,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:17:43,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:17:43,026 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:17:44,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:49,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:52,317 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:17:54,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:17:57,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:17:58,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:17:58,026 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:18:00,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:05,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:10,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:12,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:18:13,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:18:13,026 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:18:16,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:21,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:22,318 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:18:26,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:27,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:18:28,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:18:28,037 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:18:32,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:37,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:40,440 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:18:42,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:42,445 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:18:42,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:18:43,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:18:43,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:18:48,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:52,318 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:18:53,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:18:57,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:18:58,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:18:58,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:18:59,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:04,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:09,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:12,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:19:13,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:19:13,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:19:15,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:20,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:22,318 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:19:25,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:27,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:19:28,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:19:28,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:19:31,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:36,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:41,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:42,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:19:43,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:19:43,035 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:19:46,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:51,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:52,319 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:19:56,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:19:57,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:19:58,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:19:58,035 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:20:02,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:07,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:12,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:12,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:20:13,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:20:13,035 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:20:17,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:22,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:22,319 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:20:27,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:27,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:20:28,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:20:28,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:20:33,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:38,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:42,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:20:43,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:20:43,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:20:44,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:49,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:52,320 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:20:54,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:20:57,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:20:58,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:20:58,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:21:00,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:05,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:10,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:12,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:21:13,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:21:13,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:21:16,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:21,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:22,323 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:21:26,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:27,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:21:28,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:21:28,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:21:32,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:37,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:42,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:42,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:21:43,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:21:43,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:21:47,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:52,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:52,324 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:21:57,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:21:57,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:21:58,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:21:58,037 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:22:03,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:08,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:12,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:22:13,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:22:13,046 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:22:14,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:19,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:22,324 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:22:24,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:27,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:22:28,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:22:28,044 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:22:30,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:35,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:40,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:42,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:22:43,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:22:43,044 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:22:45,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:50,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:52,325 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:22:55,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:22:57,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:22:58,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:22:58,045 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:23:01,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:06,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:11,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:12,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:23:13,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:23:13,045 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:23:16,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:21,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:22,326 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:23:26,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:27,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:23:28,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:23:28,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:23:32,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:37,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:42,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:42,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:23:43,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:23:43,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:23:48,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:52,326 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:23:53,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:23:57,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:23:58,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:23:58,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:23:59,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:04,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:09,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:12,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:24:13,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:24:13,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:24:15,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:20,239 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:22,327 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:24:25,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:27,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:24:28,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:24:28,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:24:31,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:36,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:41,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:42,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:24:43,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:24:43,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:24:46,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:51,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:52,336 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:24:56,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:24:57,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:24:58,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:24:58,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:25:02,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:07,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:12,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:12,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:25:13,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:25:13,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:25:18,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:22,336 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:25:23,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:27,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:25:28,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:25:28,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:25:29,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:34,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:39,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:42,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:25:43,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:25:43,056 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:25:44,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:49,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:52,337 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:25:54,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:25:57,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:25:58,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:25:58,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:26:00,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:05,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:10,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:12,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:26:13,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:26:13,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:26:16,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:21,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:22,337 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:26:26,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:27,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:26:28,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:26:28,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:26:32,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:37,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:42,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:42,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:26:43,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:26:43,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:26:47,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:52,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:52,338 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:26:57,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:26:57,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:26:58,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:26:58,057 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:27:03,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:08,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:12,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:27:13,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:27:13,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:27:13,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:18,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:22,338 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:27:23,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:27,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:27:28,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:27:28,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:27:29,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:34,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:39,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:42,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:27:43,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:27:43,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:27:45,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:50,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:52,350 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:27:55,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:27:57,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:27:58,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:27:58,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:28:01,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:06,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:11,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:12,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:28:13,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:28:13,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:28:14,681 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:28:16,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:21,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:22,339 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:28:26,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:27,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:28:28,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:28:28,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:28:32,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:37,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:42,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:43,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:28:43,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:28:43,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:28:47,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:52,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:52,340 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:28:57,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:28:58,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:28:58,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:28:58,058 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:29:03,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:08,269 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:13,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:29:13,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:29:13,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:29:13,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:18,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:22,340 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:29:23,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:28,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:29:28,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:29:28,079 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:29:29,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:34,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:39,330 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:43,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:29:43,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:29:43,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:29:45,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:50,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:52,341 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:29:55,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:29:58,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:29:58,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:29:58,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:30:01,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:06,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:11,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:13,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:30:13,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:30:13,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:30:16,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:21,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:22,341 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:30:26,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:28,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:30:28,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:30:28,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:30:32,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:37,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:42,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:43,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:30:43,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:30:43,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:30:47,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:52,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:52,342 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:30:57,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:30:58,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:30:58,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:30:58,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:31:03,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:08,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:13,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:31:13,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:31:13,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:31:14,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:19,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:22,342 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:31:24,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:28,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:31:28,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:31:28,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:31:30,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:35,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:40,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:43,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:31:43,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:31:43,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:31:46,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:51,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:52,343 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:31:56,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:31:58,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:31:58,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:31:58,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:32:02,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:07,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:12,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:13,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:32:13,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:32:13,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:32:18,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:22,343 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:32:23,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:28,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:32:28,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:32:28,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:32:29,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:34,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:39,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:43,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:32:43,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:32:43,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:32:45,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:50,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:52,344 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:32:55,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:32:58,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:32:58,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:32:58,076 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:33:01,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:06,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:11,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:13,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:33:13,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:33:13,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:33:16,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:21,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:22,344 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:33:26,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:28,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:33:28,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:33:28,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:33:32,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:37,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:42,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:43,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:33:43,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:33:43,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:33:48,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:52,345 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:33:53,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:33:58,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:33:58,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:33:58,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:33:59,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:04,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:09,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:13,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:34:13,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:34:13,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:34:14,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:19,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:22,345 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:34:24,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:28,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:34:28,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:34:28,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:34:30,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:35,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:40,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:43,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:34:43,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:34:43,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:34:46,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:51,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:52,346 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:34:56,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:34:58,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:34:58,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:34:58,070 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:35:02,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:07,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:12,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:13,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:35:13,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:35:13,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:35:18,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:22,347 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:35:23,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:28,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:35:28,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:35:28,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:35:29,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:34,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:39,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:43,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:35:43,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:35:43,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:35:44,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:49,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:52,347 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:35:54,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:35:57,461 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:35:58,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:35:58,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:35:58,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:36:00,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:05,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:10,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:13,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:36:13,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:36:13,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:36:16,244 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:21,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:22,348 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:36:26,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:28,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:36:28,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:36:28,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:36:32,264 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:37,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:42,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:43,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:36:43,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:36:43,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:36:48,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:52,348 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:36:53,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:36:58,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:36:58,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:36:58,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:36:59,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:04,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:09,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:13,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:37:13,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:37:13,076 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:37:15,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:20,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:22,356 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:37:25,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:28,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:37:28,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:37:28,077 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:37:31,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:36,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:41,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:43,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:37:43,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:37:43,076 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:37:46,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:51,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:52,357 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:37:56,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:37:58,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:37:58,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:37:58,077 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:38:02,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:07,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:12,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:13,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:38:13,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:38:13,077 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:38:18,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:22,357 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:38:23,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:28,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:38:28,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:38:28,077 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:38:29,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:34,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:39,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:43,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:38:43,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:38:43,085 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:38:45,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:50,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:52,357 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:38:55,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:38:58,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:38:58,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:38:58,085 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:39:01,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:06,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:11,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:13,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:39:13,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:39:13,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:39:17,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:22,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:22,358 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:39:27,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:28,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:39:28,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:39:28,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:39:32,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:37,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:42,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:43,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:39:43,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:39:43,095 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:39:48,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:52,358 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:39:53,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:39:58,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:39:58,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:39:58,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:39:59,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:04,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:09,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:13,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:40:13,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:40:13,095 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:40:15,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:20,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:22,359 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:40:25,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:28,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:40:28,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:40:28,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:40:31,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:36,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:41,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:43,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:40:43,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:40:43,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:40:46,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:51,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:52,359 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:40:56,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:40:58,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:40:58,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:40:58,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:41:02,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:03,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:41:05,637 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:41:07,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:12,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:13,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:41:13,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:41:13,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:41:18,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:22,360 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:41:23,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:28,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:41:28,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:41:28,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:41:29,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:34,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:39,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:43,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:41:43,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:41:43,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:41:44,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:49,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:52,360 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:41:54,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:41:58,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:41:58,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:41:58,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:42:00,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:05,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:10,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:13,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:42:13,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:42:13,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:42:15,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:20,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:22,361 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:42:25,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:28,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:42:28,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:42:28,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:42:31,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:36,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:41,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:43,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:42:43,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:42:43,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:42:46,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:51,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:52,371 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:42:56,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:42:58,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:42:58,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:42:58,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:43:02,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:07,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:12,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:13,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:43:13,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:43:13,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:43:18,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:22,362 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:43:23,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:28,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:43:28,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:43:28,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:43:29,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:34,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:39,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:43,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:43:43,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:43:43,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:43:44,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:49,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:52,362 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:43:54,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:43:58,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:43:58,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:43:58,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:44:00,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:05,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:10,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:13,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:44:13,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:44:13,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:44:16,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:21,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:22,363 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:44:26,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:28,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:44:28,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:44:28,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:44:31,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:36,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:41,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:43,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:44:43,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:44:43,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:44:47,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:52,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:52,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:44:57,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:44:58,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:44:58,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:44:58,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:45:03,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:08,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:13,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:45:13,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:45:13,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:45:13,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:18,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:22,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:45:23,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:28,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:45:28,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:45:28,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:45:29,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:34,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:39,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:43,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:45:43,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:45:43,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:45:44,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:49,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:52,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:45:54,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:45:58,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:45:58,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:45:58,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:46:00,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:05,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:10,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:13,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:46:13,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:46:13,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:46:16,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:21,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:22,365 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:46:26,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:28,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:46:28,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:46:28,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:46:32,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:37,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:42,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:43,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:46:43,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:46:43,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:46:48,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:52,365 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:46:53,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:46:58,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:46:58,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:46:58,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:46:59,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:04,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:09,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:13,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:47:13,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:47:13,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:47:15,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:20,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:22,366 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:47:25,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:28,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:47:28,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:47:28,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:47:31,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:36,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:41,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:43,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:47:43,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:47:43,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:47:47,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:52,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:52,367 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:47:57,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:47:58,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:47:58,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:47:58,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:48:03,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:08,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:13,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:48:13,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:48:13,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:48:14,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:19,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:22,367 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:48:24,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:28,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:48:28,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:48:28,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:48:30,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:35,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:40,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:43,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:48:43,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:48:43,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:48:46,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:51,269 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:52,367 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:48:56,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:48:58,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:48:58,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:48:58,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:49:02,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:07,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:12,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:13,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:49:13,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:49:13,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:49:17,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:22,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:22,368 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:49:27,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:28,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:49:28,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:49:28,107 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:49:33,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:38,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:43,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:49:43,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:49:43,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:49:43,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:48,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:52,368 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:49:53,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:49:58,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:49:58,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:49:58,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:49:59,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:04,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:09,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:13,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:50:13,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:50:13,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:50:15,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:20,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:22,369 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:50:25,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:28,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:50:28,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:50:28,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:50:31,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:36,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:41,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:43,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:50:43,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:50:43,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:50:47,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:52,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:52,370 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:50:57,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:50:58,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:50:58,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:50:58,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:51:02,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:07,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:12,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:13,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:51:13,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:51:13,108 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:51:18,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:22,370 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:51:23,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:28,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:51:28,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:51:28,109 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:51:28,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:33,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:38,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:43,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:51:43,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:51:43,109 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:51:44,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:49,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:52,371 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:51:54,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:51:58,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:51:58,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:51:58,109 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:52:00,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:05,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:10,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:13,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:52:13,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:52:13,115 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:52:16,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:21,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:22,371 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:52:26,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:28,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:52:28,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:52:28,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:52:32,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:37,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:42,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:43,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:52:43,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:52:43,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:52:48,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:52,372 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:52:53,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:52:58,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:52:58,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:52:58,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:52:59,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:04,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:09,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:13,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:53:13,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:53:13,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:53:14,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:19,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:22,372 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:53:24,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:28,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:53:28,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:53:28,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:53:30,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:35,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:36,501 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:53:40,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:43,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:53:43,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:53:43,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:53:46,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:51,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:52,373 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:53:56,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:53:58,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:53:58,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:53:58,116 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:54:02,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:07,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:12,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:13,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:54:13,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:54:13,122 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:54:16,622 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 10:54:18,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:22,373 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:54:23,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:28,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:54:28,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:54:28,122 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:54:29,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:34,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:39,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:43,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:54:43,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:54:43,123 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:54:45,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:50,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:52,374 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:54:55,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:54:58,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:54:58,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:54:58,122 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:55:01,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:06,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:11,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:13,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:55:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:55:13,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:55:16,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:21,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:22,375 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:55:27,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:28,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:55:28,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:55:28,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:55:33,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:38,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:43,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:55:43,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:55:43,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:55:43,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:48,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:52,375 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:55:54,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:55:58,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:55:58,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:55:58,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:55:59,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:04,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:09,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:13,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:56:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:56:13,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:56:15,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:20,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:22,375 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:56:25,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:28,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:56:28,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:56:28,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:56:31,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:36,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:41,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:43,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:56:43,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:56:43,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:56:46,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:51,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:52,376 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:56:57,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:56:58,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:56:58,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:56:58,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:57:03,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:08,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:13,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:57:13,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:57:13,129 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:57:13,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:18,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:22,376 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:57:24,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:28,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:57:28,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:57:28,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:57:30,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:35,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:40,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:43,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:57:43,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:57:43,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:57:45,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:50,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:52,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:57:55,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:57:58,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:57:58,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:57:58,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:58:01,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:06,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:11,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:13,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:58:13,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:58:13,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:58:16,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:21,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:22,377 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:58:26,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:28,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:58:28,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:58:28,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:58:32,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:37,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:42,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:43,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:58:43,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:58:43,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:58:48,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:52,378 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:58:53,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:58:58,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:58:58,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:58:58,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:58:59,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:04,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:09,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:13,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:59:13,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:59:13,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:59:15,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:20,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:22,378 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:59:25,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:28,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:59:28,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:59:28,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:59:30,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:35,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:40,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:43,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:59:43,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:59:43,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 10:59:46,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:51,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:52,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 10:59:56,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 10:59:58,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 10:59:58,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 10:59:58,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:00:02,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:07,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:12,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:13,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:00:13,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:00:13,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:00:17,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:22,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:00:23,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:28,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:00:28,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:00:28,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:00:28,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:33,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:38,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:43,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:00:43,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:00:43,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:00:43,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:48,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:52,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:00:54,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:00:58,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:00:58,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:00:58,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:01:00,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:05,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:10,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:13,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:01:13,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:01:13,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:01:15,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:20,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:22,380 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:01:25,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:28,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:01:28,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:01:28,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:01:31,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:36,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:41,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:43,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:01:43,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:01:43,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:01:47,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:52,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:52,380 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:01:57,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:01:58,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:01:58,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:01:58,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:02:03,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:08,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:13,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:02:13,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:02:13,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:02:13,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:18,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:22,381 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:02:23,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:28,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:02:28,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:02:28,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:02:28,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:33,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:38,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:43,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:02:43,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:02:43,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:02:44,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:49,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:52,381 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:02:55,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:02:58,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:02:58,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:02:58,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:03:01,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:06,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:11,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:13,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:03:13,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:03:13,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:03:16,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:21,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:22,382 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:03:27,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:28,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:03:28,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:03:28,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:03:33,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:38,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:43,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:03:43,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:03:43,134 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:03:43,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:48,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:52,382 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:03:54,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:03:58,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:03:58,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:03:58,132 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:03:59,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:04,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:09,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:13,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:04:13,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:04:13,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:04:15,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:20,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:22,383 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:04:26,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:28,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:04:28,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:04:28,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:04:31,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:36,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:41,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:43,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:04:43,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:04:43,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:04:46,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:51,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:52,383 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:04:57,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:04:58,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:04:58,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:04:58,142 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:05:03,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:08,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:13,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:05:13,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:05:13,150 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:05:13,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:18,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:22,383 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:05:23,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:28,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:05:28,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:05:28,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:05:29,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:34,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:39,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:43,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:05:43,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:05:43,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:05:45,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:50,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:52,384 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:05:55,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:05:58,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:05:58,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:05:58,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:06:01,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:06,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:11,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:13,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:06:13,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:06:13,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:06:17,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:22,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:22,384 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:06:27,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:28,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:06:28,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:06:28,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:06:33,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:38,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:43,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:06:43,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:06:43,152 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:06:43,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:48,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:52,386 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:06:53,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:58,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:06:58,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:06:58,144 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:06:59,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:06:59,342 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:07:04,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:09,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:13,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:07:13,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:07:13,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:07:14,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:19,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:22,386 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:07:24,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:28,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:07:28,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:07:28,143 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:07:30,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:35,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:40,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:43,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:07:43,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:07:43,144 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:07:45,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:50,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:52,386 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:07:55,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:07:58,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:07:58,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:07:58,144 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:08:01,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:06,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:11,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:13,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:08:13,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:08:13,146 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:08:17,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:22,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:22,387 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:08:27,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:28,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:08:28,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:08:28,146 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:08:33,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:38,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:43,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:08:43,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:08:43,146 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:08:44,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:49,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:52,397 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:08:54,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:08:58,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:08:58,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:08:58,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:09:00,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:05,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:10,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:13,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:09:13,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:09:13,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:09:15,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:20,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:22,387 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:09:25,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:28,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:09:28,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:09:28,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:09:31,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:36,330 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:41,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:43,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:09:43,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:09:43,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:09:46,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:51,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:52,388 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:09:56,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:09:58,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:09:58,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:09:58,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:10:02,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:07,068 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:10:07,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:12,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:13,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:10:13,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:10:13,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:10:18,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:22,388 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:10:23,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:28,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:10:28,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:10:28,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:10:29,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:34,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:39,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:43,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:10:43,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:10:43,147 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:10:44,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:49,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:52,389 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:10:55,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:10:58,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:10:58,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:10:58,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:11:00,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:05,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:10,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:13,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:11:13,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:11:13,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:11:16,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:21,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:22,389 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:11:26,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:28,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:11:28,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:11:28,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:11:31,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:36,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:41,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:43,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:11:43,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:11:43,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:11:47,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:52,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:52,390 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:11:57,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:11:58,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:11:58,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:11:58,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:12:02,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:07,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:12,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:13,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:12:13,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:12:13,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:12:18,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:22,390 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:12:23,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:28,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:12:28,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:12:28,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:12:28,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:33,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:38,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:43,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:12:43,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:12:43,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:12:44,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:49,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:52,391 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:12:54,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:12:58,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:12:58,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:12:58,157 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:13:00,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:05,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:10,330 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:13,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:13:13,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:13:13,157 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:13:16,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:21,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:22,401 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:13:26,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:28,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:13:28,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:13:28,157 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:13:32,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:37,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:42,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:43,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:13:43,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:13:43,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:13:48,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:52,392 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:13:53,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:13:58,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:13:58,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:13:58,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:13:59,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:04,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:09,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:13,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:14:13,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:14:13,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:14:14,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:19,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:22,392 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:14:24,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:28,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:14:28,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:14:28,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:14:29,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:34,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:39,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:43,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:14:43,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:14:43,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:14:45,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:50,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:52,393 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:14:55,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:14:58,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:14:58,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:14:58,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:15:00,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:05,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:10,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:13,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:15:13,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:15:13,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:15:16,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:21,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:22,394 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:15:26,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:28,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:15:28,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:15:28,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:15:32,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:37,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:42,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:43,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:15:43,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:15:43,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:15:47,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:52,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:52,394 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:15:57,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:15:58,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:15:58,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:15:58,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:16:02,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:07,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:12,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:13,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:16:13,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:16:13,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:16:18,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:22,394 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:16:23,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:28,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:16:28,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:16:28,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:16:28,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:33,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:38,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:43,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:16:43,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:16:43,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:16:44,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:49,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:52,395 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:16:54,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:16:58,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:16:58,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:16:58,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:17:00,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:05,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:10,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:13,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:17:13,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:17:13,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:17:16,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:21,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:22,395 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:17:26,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:28,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:17:28,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:17:28,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:17:32,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:37,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:42,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:43,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:17:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:17:43,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:17:48,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:52,396 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:17:53,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:17:58,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:17:58,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:17:58,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:17:59,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:04,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:09,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:13,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:18:13,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:18:13,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:18:15,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:20,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:22,396 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:18:25,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:28,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:18:28,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:18:28,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:18:30,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:35,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:40,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:43,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:18:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:18:43,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:18:46,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:51,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:52,397 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:18:56,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:18:58,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:18:58,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:18:58,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:19:02,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:07,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:12,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:13,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:19:13,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:19:13,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:19:17,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:22,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:22,397 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:19:27,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:28,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:19:28,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:19:28,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:19:32,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:37,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:42,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:43,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:19:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:19:43,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:19:48,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:52,398 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:19:53,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:19:58,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:19:58,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:19:58,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:19:59,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:04,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:09,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:13,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:20:13,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:20:13,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:20:15,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:21,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:22,398 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:20:22,483 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:20:26,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:28,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:20:28,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:20:28,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:20:32,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:37,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:42,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:43,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:20:43,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:20:43,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:20:47,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:52,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:52,399 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:20:57,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:20:58,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:20:58,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:20:58,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:21:03,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:08,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:13,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:21:13,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:21:13,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:21:13,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:18,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:22,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:21:24,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:28,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:21:28,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:21:28,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:21:30,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:35,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:40,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:43,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:21:43,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:21:43,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:21:46,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:51,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:52,399 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:21:56,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:21:58,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:21:58,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:21:58,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:22:02,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:07,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:12,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:13,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:22:13,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:22:13,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:22:17,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:22,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:22:23,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:28,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:22:28,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:22:28,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:22:29,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:34,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:39,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:43,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:22:43,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:22:43,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:22:45,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:50,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:52,401 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:22:55,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:22:58,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:22:58,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:22:58,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:23:00,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:05,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:10,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:13,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:23:13,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:23:13,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:23:16,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:21,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:22,401 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:23:26,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:28,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:23:28,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:23:28,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:23:32,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:37,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:42,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:43,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:23:43,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:23:43,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:23:47,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:52,402 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:23:53,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:23:58,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:23:58,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:23:58,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:23:59,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:04,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:09,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:13,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:24:13,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:24:13,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:24:15,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:20,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:22,403 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:24:25,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:28,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:24:28,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:24:28,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:24:31,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:36,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:41,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:43,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:24:43,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:24:43,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:24:46,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:51,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:52,403 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:24:57,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:24:58,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:24:58,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:24:58,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:25:03,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:08,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:13,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:25:13,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:25:13,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:25:14,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:19,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:22,404 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:25:24,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:28,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:25:28,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:25:28,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:25:30,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:35,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:40,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:43,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:25:43,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:25:43,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:25:45,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:50,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:52,404 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:25:56,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:25:58,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:25:58,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:25:58,170 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:26:02,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:07,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:12,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:13,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:26:13,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:26:13,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:26:17,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:22,405 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:26:23,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:28,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:26:28,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:26:28,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:26:29,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:34,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:39,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:43,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:26:43,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:26:43,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:26:45,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:50,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:52,405 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:26:55,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:26:58,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:26:58,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:26:58,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:27:00,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:05,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:10,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:13,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:27:13,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:27:13,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:27:16,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:18,921 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:27:21,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:22,406 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:27:27,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:28,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:27:28,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:27:28,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:27:33,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:38,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:43,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:27:43,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:27:43,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:27:44,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:49,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:52,406 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:27:54,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:27:58,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:27:58,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:27:58,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:28:00,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:05,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:10,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:13,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:28:13,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:28:13,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:28:16,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:21,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:22,417 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:28:26,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:28,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:28:28,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:28:28,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:28:32,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:37,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:42,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:43,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:28:43,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:28:43,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:28:47,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:52,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:52,407 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:28:57,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:28:58,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:28:58,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:28:58,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:29:02,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:07,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:12,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:13,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:29:13,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:29:13,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:29:18,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:22,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:29:24,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:28,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:29:28,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:29:28,181 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:29:30,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:35,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:40,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:43,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:29:43,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:29:43,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:29:46,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:51,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:52,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:29:56,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:29:58,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:29:58,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:29:58,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:30:02,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:07,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:12,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:13,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:30:13,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:30:13,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:30:18,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:22,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:30:23,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:28,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:30:28,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:30:28,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:30:28,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:33,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:38,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:43,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:30:43,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:30:43,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:30:43,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:48,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:52,410 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:30:54,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:30:58,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:30:58,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:30:58,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:30:59,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:04,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:09,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:13,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:31:13,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:31:13,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:31:15,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:20,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:22,410 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:31:25,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:28,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:31:28,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:31:28,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:31:31,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:36,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:41,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:43,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:31:43,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:31:43,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:31:47,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:52,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:52,411 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:31:57,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:31:58,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:31:58,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:31:58,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:32:02,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:07,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:12,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:13,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:32:13,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:32:13,182 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:32:18,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:22,411 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:32:23,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:28,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:32:28,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:32:28,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:32:28,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:33,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:38,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:43,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:32:43,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:32:43,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:32:44,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:49,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:52,411 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:32:54,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:32:58,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:32:58,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:32:58,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:32:59,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:04,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:07,253 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:33:09,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:13,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:33:13,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:33:13,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:33:15,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:20,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:22,412 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:33:25,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:28,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:33:28,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:33:28,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:33:31,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:36,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:41,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:43,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:33:43,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:33:43,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:33:46,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:51,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:52,412 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:33:57,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:33:58,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:33:58,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:33:58,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:34:03,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:08,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:13,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:34:13,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:34:13,183 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:34:13,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:18,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:22,413 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:34:24,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:28,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:34:28,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:34:28,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:34:29,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:34,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:39,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:43,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:34:43,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:34:43,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:34:45,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:50,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:52,414 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:34:56,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:34:58,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:34:58,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:34:58,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:35:01,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:06,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:11,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:13,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:35:13,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:35:13,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:35:17,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:22,414 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:35:23,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:28,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:35:28,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:35:28,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:35:29,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:34,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:39,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:43,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:35:43,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:35:43,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:35:44,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:49,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:52,415 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:35:55,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:35:58,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:35:58,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:35:58,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:36:01,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:06,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:11,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:13,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:36:13,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:36:13,184 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:36:17,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:22,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:22,415 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:36:27,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:28,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:36:28,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:36:28,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:36:32,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:37,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:42,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:43,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:36:43,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:36:43,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:36:48,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:52,416 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:36:54,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:36:58,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:36:58,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:36:58,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:37:00,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:05,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:10,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:13,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:37:13,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:37:13,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:37:16,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:21,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:22,416 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:37:26,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:28,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:37:28,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:37:28,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:37:31,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:36,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:41,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:43,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:37:43,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:37:43,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:37:47,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:52,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:52,418 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:37:57,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:37:58,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:37:58,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:37:58,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:38:03,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:08,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:13,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:38:13,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:38:13,185 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:38:14,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:19,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:22,417 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:38:24,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:28,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:38:28,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:38:28,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:38:30,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:35,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:40,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:43,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:38:43,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:38:43,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:38:45,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:50,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:52,418 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:38:55,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:38:58,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:38:58,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:38:58,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:39:01,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:06,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:11,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:39:13,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:39:13,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:39:16,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:21,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:22,418 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:39:26,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:28,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:39:28,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:39:28,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:39:31,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:36,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:41,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:43,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:39:43,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:39:43,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:39:47,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:52,419 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:39:53,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:39:58,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:39:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:39:58,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:39:59,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:04,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:09,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:40:13,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:40:13,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:40:14,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:19,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:22,419 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:40:24,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:28,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:40:28,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:40:28,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:40:29,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:34,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:39,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:43,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:40:43,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:40:43,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:40:45,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:50,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:52,424 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:40:55,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:40:58,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:40:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:40:58,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:41:00,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:05,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:10,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:41:13,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:41:13,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:41:16,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:21,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:22,425 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:41:26,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:28,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:41:28,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:41:28,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:41:31,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:36,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:41,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:43,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:41:43,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:41:43,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:41:47,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:52,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:52,425 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:41:57,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:41:58,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:41:58,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:41:58,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:42:03,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:08,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:13,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:42:13,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:42:13,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:42:13,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:18,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:22,426 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:42:23,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:28,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:42:28,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:42:28,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:42:29,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:34,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:39,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:43,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:42:43,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:42:43,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:42:45,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:50,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:52,426 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:42:55,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:42:58,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:42:58,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:42:58,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:43:01,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:06,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:11,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:13,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:43:13,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:43:13,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:43:17,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:22,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:22,427 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:43:27,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:28,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:43:28,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:43:28,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:43:33,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:38,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:43,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:43:43,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:43:43,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:43:44,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:49,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:52,438 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:43:54,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:43:58,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:43:58,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:43:58,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:44:00,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:05,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:10,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:13,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:44:13,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:44:13,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:44:15,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:20,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:22,428 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:44:26,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:28,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:44:28,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:44:28,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:44:32,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:37,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:39,787 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:44:43,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:43,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:44:43,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:44:43,189 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:44:48,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:52,428 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:44:53,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:44:58,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:44:58,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:44:58,193 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:44:59,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:04,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:09,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:13,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:45:13,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:45:13,189 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:45:14,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:19,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:22,429 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:45:24,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:28,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:45:28,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:45:28,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:45:30,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:35,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:40,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:43,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:45:43,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:45:43,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:45:45,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:50,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:52,429 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:45:56,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:45:58,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:45:58,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:45:58,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:46:01,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:04,108 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:46:06,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:11,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:13,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:46:13,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:46:13,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:46:17,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:22,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:22,430 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:46:27,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:28,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:46:28,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:46:28,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:46:33,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:38,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:43,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:46:43,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:46:43,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:46:43,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:48,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:52,430 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:46:54,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:46:58,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:46:58,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:46:58,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:47:00,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:05,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:10,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:13,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:47:13,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:47:13,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:47:16,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:21,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:22,431 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:47:26,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:28,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:47:28,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:47:28,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:47:32,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:37,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:42,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:43,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:47:43,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:47:43,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:47:48,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:52,431 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:47:53,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:47:58,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:47:58,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:47:58,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:47:59,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:04,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:09,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:13,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:48:13,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:48:13,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:48:15,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:20,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:22,432 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:48:25,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:28,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:48:28,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:48:28,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:48:31,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:36,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:41,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:43,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:48:43,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:48:43,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:48:46,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:51,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:52,432 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:48:57,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:48:58,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:48:58,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:48:58,197 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:49:03,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:08,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:13,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:49:13,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:49:13,200 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:49:14,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:19,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:22,433 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:49:24,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:28,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:49:28,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:49:28,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:49:29,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:34,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:39,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:43,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:49:43,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:49:43,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:49:45,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:50,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:52,433 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:49:55,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:49:58,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:49:58,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:49:58,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:50:01,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:06,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:11,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:13,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:50:13,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:50:13,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:50:16,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:21,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:22,434 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:50:26,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:28,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:50:28,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:50:28,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:50:32,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:37,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:42,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:43,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:50:43,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:50:43,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:50:48,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:52,434 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:50:53,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:50:58,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:50:58,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:50:58,198 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:50:58,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:03,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:08,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:13,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:51:13,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:51:13,200 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:51:14,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:19,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:22,435 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:51:24,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:28,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:51:28,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:51:28,209 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:51:29,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:34,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:39,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:43,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:51:43,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:51:43,209 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:51:45,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:50,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:52,436 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:51:55,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:51:58,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:51:58,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:51:58,209 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:52:00,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:05,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:10,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:13,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:52:13,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:52:13,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:52:16,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:21,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:22,436 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:52:27,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:28,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:52:28,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:52:28,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:52:33,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:38,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:43,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:52:43,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:52:43,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:52:43,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:48,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:52,436 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:52:54,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:52:58,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:52:58,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:52:58,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:53:00,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:05,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:10,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:13,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:53:13,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:53:13,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:53:15,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:20,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:22,437 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:53:26,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:28,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:53:28,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:53:28,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:53:31,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:36,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:41,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:43,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:53:43,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:53:43,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:53:47,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:52,437 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:53:52,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:57,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:53:58,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:53:58,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:53:58,211 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:54:02,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:07,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:12,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:13,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:54:13,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:54:13,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:54:18,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:22,438 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:54:24,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:28,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:54:28,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:54:28,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:54:29,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:34,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:39,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:43,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:54:43,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:54:43,213 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:54:45,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:50,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:52,438 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:54:55,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:54:58,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:54:58,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:54:58,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:55:01,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:06,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:11,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:13,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:55:13,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:55:13,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:55:16,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:21,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:22,439 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:55:27,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:28,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:55:28,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:55:28,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:55:33,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:38,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:43,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:55:43,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:55:43,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:55:43,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:48,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:52,439 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:55:54,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:55:58,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:55:58,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:55:58,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:56:00,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:05,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:10,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:13,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:56:13,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:56:13,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:56:16,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:21,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:22,445 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:56:26,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:28,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:56:28,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:56:28,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:56:32,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:37,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:42,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:43,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:56:43,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:56:43,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:56:47,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:52,440 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:56:53,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:56:58,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:56:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:56:58,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:56:59,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:04,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:09,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:13,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:57:13,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:57:13,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:57:15,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:20,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:22,441 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:57:25,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:28,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:57:28,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:57:28,222 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:57:31,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:36,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:41,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:43,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:57:43,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:57:43,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:57:46,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:51,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:52,442 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:57:57,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:57:58,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:57:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:57:58,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:58:03,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:08,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:13,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:58:13,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:58:13,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:58:13,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:18,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:22,442 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:58:24,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:28,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:58:28,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:58:28,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:58:30,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:35,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:40,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:43,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:58:43,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:58:43,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:58:45,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:50,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:52,443 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:58:56,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:58:58,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:58:58,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:58:58,226 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:59:01,068 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 11:59:01,842 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:06,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:11,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:13,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:59:13,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:59:13,224 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:59:17,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:22,443 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:59:23,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:28,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:59:28,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:59:28,224 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:59:28,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:33,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:38,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:43,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:59:43,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:59:43,224 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:59:44,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:49,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:52,444 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 11:59:54,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 11:59:58,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 11:59:58,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 11:59:58,224 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 11:59:59,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:04,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:09,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:13,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:00:13,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:00:13,224 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:00:15,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:20,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:22,444 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:00:26,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:28,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:00:28,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:00:28,230 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:00:31,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:36,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:41,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:43,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:00:43,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:00:43,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:00:47,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:52,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:52,445 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:00:57,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:00:58,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:00:58,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:00:58,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:01:02,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:07,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:12,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:13,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:01:13,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:01:13,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:01:18,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:22,445 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:01:24,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:28,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:01:28,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:01:28,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:01:29,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:34,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:39,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:43,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:01:43,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:01:43,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:01:45,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:50,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:52,446 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:01:55,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:01:58,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:01:58,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:01:58,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:02:00,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:05,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:08,818 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:02:10,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:13,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:02:13,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:02:13,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:02:16,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:21,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:22,446 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:02:26,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:28,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:02:28,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:02:28,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:02:32,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:37,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:42,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:43,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:02:43,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:02:43,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:02:47,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:52,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:52,447 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:02:57,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:02:58,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:02:58,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:02:58,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:03:02,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:07,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:12,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:13,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:03:13,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:03:13,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:03:18,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:22,466 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:03:24,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:28,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:03:28,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:03:28,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:03:30,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:35,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:40,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:43,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:03:43,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:03:43,241 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:03:45,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:50,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:52,466 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:03:55,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:03:58,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:03:58,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:03:58,241 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:04:01,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:06,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:11,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:13,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:04:13,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:04:13,241 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:04:17,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:22,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:22,467 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:04:27,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:28,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:04:28,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:04:28,241 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:04:33,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:38,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:04:43,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:04:43,242 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:04:43,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:48,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:52,467 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:04:53,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:04:58,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:04:58,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:04:58,242 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:04:58,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:03,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:08,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:13,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:05:13,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:05:13,242 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:05:14,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:19,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:22,468 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:05:24,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:28,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:05:28,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:05:28,242 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:05:30,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:35,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:40,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:05:43,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:05:43,242 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:05:45,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:50,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:52,468 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:05:56,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:05:58,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:05:58,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:05:58,252 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:06:02,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:07,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:12,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:13,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:06:13,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:06:13,252 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:06:17,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:22,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:22,469 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:06:27,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:28,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:06:28,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:06:28,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:06:33,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:38,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:43,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:06:43,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:06:43,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:06:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:48,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:52,469 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:06:53,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:06:58,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:06:58,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:06:58,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:06:59,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:04,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:09,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:13,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:07:13,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:07:13,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:07:14,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:19,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:22,469 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:07:25,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:28,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:07:28,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:07:28,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:07:31,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:36,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:41,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:43,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:07:43,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:07:43,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:07:46,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:51,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:52,470 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:07:57,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:07:58,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:07:58,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:07:58,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:08:03,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:08,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:13,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:08:13,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:08:13,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:08:13,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:18,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:22,471 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:08:23,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:28,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:08:28,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:08:28,263 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:08:29,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:34,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:39,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:43,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:08:43,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:08:43,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:08:44,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:49,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:52,471 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:08:55,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:08:58,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:08:58,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:08:58,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:09:01,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:06,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:11,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:13,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:09:13,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:09:13,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:09:16,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:21,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:22,471 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:09:27,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:28,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:09:28,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:09:28,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:09:33,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:38,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:43,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:09:43,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:09:43,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:09:44,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:49,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:52,472 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:09:55,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:09:58,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:09:58,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:09:58,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:10:00,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:05,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:10,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:13,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:10:13,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:10:13,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:10:15,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:20,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:22,472 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:10:26,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:28,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:10:28,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:10:28,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:10:31,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:36,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:41,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:43,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:10:43,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:10:43,283 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:10:46,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:51,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:52,473 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:10:57,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:10:58,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:10:58,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:10:58,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:11:03,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:08,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:13,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:11:13,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:11:13,283 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:11:13,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:18,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:22,473 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:11:24,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:28,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:11:28,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:11:28,286 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:11:29,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:34,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:39,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:43,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:11:43,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:11:43,304 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:11:44,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:49,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:52,474 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:11:55,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:11:58,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:11:58,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:11:58,298 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:12:00,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:05,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:10,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:13,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:12:13,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:12:13,298 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:12:15,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:20,984 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:12:21,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:22,474 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:12:26,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:28,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:12:28,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:12:28,297 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:12:31,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:36,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:41,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:43,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:12:43,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:12:43,298 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:12:46,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:51,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:52,475 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:12:57,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:12:58,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:12:58,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:12:58,298 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:13:02,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:07,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:12,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:13,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:13:13,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:13:13,298 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:13:18,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:22,476 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:13:24,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:28,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:13:28,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:13:28,299 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:13:29,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:34,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:39,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:43,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:13:43,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:13:43,299 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:13:44,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:49,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:52,486 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:13:55,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:13:58,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:13:58,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:13:58,299 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:14:00,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:05,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:10,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:13,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:14:13,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:14:13,299 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:14:15,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:20,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:22,476 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:14:26,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:28,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:14:28,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:14:28,301 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:14:31,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:36,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:41,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:43,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:14:43,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:14:43,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:14:47,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:52,478 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:14:53,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:14:58,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:14:58,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:14:58,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:14:58,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:03,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:08,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:13,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:15:13,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:15:13,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:15:13,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:18,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:22,477 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:15:24,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:28,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:15:28,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:15:28,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:15:30,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:35,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:40,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:43,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:15:43,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:15:43,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:15:45,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:50,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:52,486 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:15:56,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:15:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:15:58,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:15:58,302 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:16:01,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:06,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:11,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:13,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:16:13,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:16:13,303 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:16:17,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:22,486 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:16:23,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:28,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:16:28,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:16:28,303 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:16:28,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:33,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:38,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:43,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:16:43,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:16:43,303 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:16:43,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:48,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:52,487 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:16:54,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:16:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:16:58,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:16:58,303 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:16:59,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:04,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:09,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:13,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:17:13,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:17:13,303 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:17:15,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:20,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:22,487 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:17:26,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:28,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:17:28,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:17:28,336 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:17:31,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:36,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:41,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:43,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:17:43,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:17:43,336 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:17:47,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:52,487 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:17:53,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:17:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:17:58,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:17:58,336 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:17:58,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:03,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:08,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:13,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:18:13,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:18:13,337 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:18:13,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:18,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:22,488 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:18:24,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:28,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:18:28,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:18:28,336 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:18:29,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:34,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:39,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:43,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:18:43,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:18:43,337 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:18:43,483 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:18:45,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:50,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:52,489 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:18:56,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:18:58,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:18:58,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:18:58,337 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:19:01,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:06,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:11,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:13,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:19:13,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:19:13,337 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:19:17,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:22,489 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:19:23,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:28,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:19:28,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:19:28,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:19:28,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:33,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:38,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:43,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:19:43,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:19:43,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:19:44,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:49,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:52,489 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:19:55,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:19:58,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:19:58,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:19:58,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:20:00,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:05,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:10,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:13,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:20:13,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:20:13,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:20:16,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:21,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:22,490 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:20:27,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:28,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:20:28,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:20:28,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:20:32,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:37,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:42,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:43,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:20:43,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:20:43,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:20:47,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:52,491 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:20:53,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:20:58,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:20:58,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:20:58,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:20:58,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:03,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:08,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:13,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:21:13,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:21:13,338 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:21:13,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:18,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:22,492 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:21:24,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:28,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:21:28,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:21:28,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:21:29,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:34,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:39,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:43,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:21:43,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:21:43,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:21:45,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:50,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:52,492 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:21:56,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:21:58,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:21:58,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:21:58,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:22:01,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:06,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:11,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:13,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:22:13,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:22:13,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:22:17,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:22,501 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:22:23,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:28,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:22:28,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:22:28,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:22:28,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:33,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:38,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:43,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:22:43,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:22:43,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:22:44,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:49,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:52,501 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:22:55,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:22:58,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:22:58,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:22:58,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:23:00,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:05,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:10,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:13,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:23:13,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:23:13,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:23:15,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:20,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:22,502 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:23:26,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:28,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:23:28,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:23:28,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:23:31,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:36,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:41,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:43,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:23:43,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:23:43,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:23:46,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:51,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:52,502 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:23:57,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:23:58,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:23:58,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:23:58,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:24:02,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:07,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:12,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:13,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:24:13,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:24:13,346 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:24:17,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:22,503 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:24:23,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:28,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:24:28,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:24:28,346 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:24:28,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:33,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:38,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:43,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:24:43,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:24:43,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:24:44,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:49,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:52,503 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:24:55,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:24:58,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:24:58,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:24:58,346 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:25:00,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:05,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:10,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:13,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:25:13,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:25:13,346 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:25:16,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:21,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:22,504 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:25:27,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:28,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:25:28,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:25:28,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:25:32,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:37,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:42,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:43,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:25:43,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:25:43,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:25:48,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:48,026 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:25:52,504 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:25:53,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:25:58,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:25:58,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:25:58,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:25:58,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:03,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:08,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:13,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:26:13,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:26:13,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:26:14,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:19,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:22,505 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:26:25,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:28,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:26:28,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:26:28,347 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:26:31,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:36,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:41,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:43,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:26:43,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:26:43,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:26:46,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:51,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:52,505 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:26:57,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:26:58,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:26:58,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:26:58,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:27:02,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:07,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:12,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:13,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:27:13,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:27:13,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:27:18,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:22,506 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:27:24,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:28,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:27:28,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:27:28,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:27:29,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:34,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:39,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:43,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:27:43,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:27:43,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:27:44,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:49,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:52,506 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:27:55,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:27:58,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:27:58,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:27:58,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:28:00,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:05,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:10,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:13,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:28:13,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:28:13,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:28:15,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:20,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:22,507 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:28:26,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:28,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:28:28,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:28:28,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:28:31,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:36,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:41,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:43,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:28:43,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:28:43,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:28:46,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:51,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:52,507 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:28:57,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:28:58,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:28:58,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:28:58,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:29:02,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:07,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:12,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:13,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:29:13,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:29:13,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:29:17,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:22,509 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:29:23,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:28,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:29:28,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:29:28,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:29:28,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:33,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:38,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:43,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:29:43,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:29:43,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:29:44,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:49,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:52,508 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:29:55,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:29:58,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:29:58,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:29:58,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:30:00,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:05,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:10,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:13,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:30:13,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:30:13,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:30:15,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:20,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:22,509 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:30:26,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:28,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:30:28,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:30:28,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:30:32,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:37,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:42,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:43,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:30:43,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:30:43,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:30:47,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:52,509 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:30:53,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:30:58,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:30:58,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:30:58,351 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:30:58,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:03,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:08,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:13,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:31:13,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:31:13,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:31:14,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:19,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:22,510 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:31:25,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:28,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:31:28,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:31:28,356 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:31:30,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:35,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:40,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:43,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:31:43,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:31:43,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:31:45,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:50,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:52,510 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:31:56,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:31:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:31:58,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:31:58,356 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:32:02,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:07,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:12,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:13,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:32:13,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:32:13,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:32:17,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:22,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:32:23,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:28,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:32:28,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:32:28,356 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:32:28,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:33,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:38,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:43,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:32:43,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:32:43,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:32:44,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:49,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:52,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:32:55,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:32:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:32:58,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:32:58,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:33:00,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:05,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:10,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:13,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:33:13,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:33:13,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:33:16,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:21,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:22,512 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:33:27,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:28,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:33:28,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:33:28,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:33:32,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:37,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:42,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:43,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:33:43,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:33:43,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:33:48,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:52,512 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:33:54,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:33:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:33:58,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:33:58,358 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:33:59,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:04,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:09,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:13,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:34:13,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:34:13,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:34:14,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:19,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:22,513 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:34:25,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:28,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:34:28,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:34:28,358 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:34:30,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:35,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:40,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:43,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:34:43,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:34:43,361 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:34:46,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:51,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:52,513 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:34:57,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:34:58,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:34:58,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:34:58,375 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:35:02,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:07,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:12,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:13,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:35:13,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:35:13,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:35:18,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:22,514 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:35:24,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:28,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:35:28,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:35:28,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:35:29,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:34,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:39,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:43,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:35:43,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:35:43,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:35:45,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:50,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:52,514 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:35:56,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:35:58,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:35:58,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:35:58,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:36:01,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:05,306 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:36:06,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:11,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:13,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:36:13,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:36:13,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:36:17,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:22,515 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:36:23,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:28,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:36:28,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:36:28,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:36:28,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:33,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:38,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:43,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:36:43,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:36:43,382 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:36:44,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:49,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:52,515 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:36:55,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:36:58,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:36:58,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:36:58,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:37:00,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:05,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:10,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:13,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:37:13,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:37:13,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:37:15,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:20,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:22,516 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:37:26,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:28,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:37:28,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:37:28,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:37:31,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:36,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:41,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:43,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:37:43,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:37:43,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:37:47,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:52,516 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:37:53,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:37:58,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:37:58,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:37:58,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:37:58,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:03,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:08,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:13,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:38:13,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:38:13,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:38:14,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:19,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:22,516 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:38:25,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:28,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:38:28,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:38:28,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:38:30,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:35,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:40,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:43,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:38:43,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:38:43,383 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:38:45,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:51,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:52,517 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:38:52,967 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:38:57,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:38:58,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:38:58,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:38:58,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:39:02,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:07,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:12,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:13,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:39:13,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:39:13,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:39:17,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:22,517 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:39:23,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:28,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:39:28,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:39:28,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:39:28,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:33,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:38,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:43,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:39:43,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:39:43,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:39:44,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:49,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:52,518 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:39:55,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:39:58,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:39:58,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:39:58,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:40:00,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:05,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:10,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:13,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:40:13,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:40:13,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:40:16,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:21,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:22,518 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:40:27,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:28,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:40:28,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:40:28,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:40:32,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:37,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:42,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:43,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:40:43,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:40:43,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:40:48,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:52,520 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:40:54,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:40:58,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:40:58,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:40:58,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:40:59,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:04,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:09,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:13,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:41:13,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:41:13,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:41:14,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:19,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:22,519 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:41:25,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:28,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:41:28,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:41:28,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:41:30,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:35,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:40,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:43,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:41:43,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:41:43,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:41:46,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:51,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:52,520 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:41:57,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:41:58,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:41:58,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:41:58,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:42:02,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:07,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:12,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:13,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:42:13,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:42:13,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:42:18,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:22,525 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:42:24,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:28,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:42:28,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:42:28,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:42:29,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:34,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:39,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:43,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:42:43,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:42:43,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:42:45,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:50,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:52,521 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:42:56,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:42:58,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:42:58,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:42:58,385 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:43:01,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:06,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:11,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:13,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:43:13,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:43:13,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:43:17,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:22,522 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:43:23,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:28,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:43:28,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:43:28,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:43:28,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:33,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:38,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:43,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:43:43,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:43:43,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:43:43,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:48,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:52,522 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:43:54,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:43:58,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:43:58,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:43:58,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:43:59,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:04,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:09,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:13,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:44:13,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:44:13,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:44:15,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:20,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:22,533 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:44:26,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:28,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:44:28,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:44:28,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:44:31,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:36,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:41,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:43,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:44:43,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:44:43,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:44:46,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:51,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:52,523 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:44:57,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:44:58,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:44:58,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:44:58,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:45:02,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:07,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:12,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:13,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:45:13,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:45:13,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:45:18,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:22,524 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:45:24,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:28,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:45:28,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:45:28,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:45:29,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:34,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:39,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:43,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:45:43,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:45:43,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:45:44,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:49,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:52,533 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:45:55,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:45:58,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:45:58,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:45:58,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:46:00,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:05,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:10,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:13,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:46:13,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:46:13,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:46:15,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:20,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:22,534 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:46:26,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:28,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:46:28,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:46:28,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:46:31,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:36,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:41,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:43,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:46:43,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:46:43,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:46:46,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:51,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:52,535 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:46:57,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:46:58,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:46:58,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:46:58,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:47:02,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:07,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:12,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:13,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:47:13,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:47:13,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:47:18,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:22,535 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:47:24,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:28,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:47:28,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:47:28,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:47:29,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:34,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:39,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:43,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:47:43,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:47:43,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:47:45,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:50,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:52,536 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:47:56,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:47:58,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:47:58,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:47:58,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:48:01,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:06,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:11,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:13,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:48:13,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:48:13,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:48:16,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:21,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:22,536 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:48:27,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:28,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:48:28,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:48:28,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:48:32,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:37,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:42,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:43,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:48:43,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:48:43,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:48:47,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:52,536 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:48:53,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:48:58,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:48:58,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:48:58,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:48:58,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:03,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:08,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:13,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:49:13,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:49:13,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:49:13,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:18,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:22,537 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:49:24,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:28,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:49:28,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:49:28,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:49:29,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:34,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:39,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:43,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:49:43,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:49:43,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:49:45,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:50,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:52,537 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:49:56,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:49:58,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:49:58,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:49:58,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:50:01,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:06,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:11,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:13,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:50:13,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:50:13,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:50:16,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:21,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:22,538 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:50:27,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:28,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:50:28,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:50:28,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:50:32,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:37,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:42,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:43,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:50:43,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:50:43,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:50:47,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:52,538 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:50:53,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:50:58,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:50:58,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:50:58,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:50:58,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:03,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:08,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:13,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:51:13,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:51:13,396 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:51:14,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:19,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:22,539 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:51:25,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:28,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:51:28,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:51:28,396 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:51:30,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:35,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:40,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:43,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:51:43,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:51:43,396 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:51:45,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:50,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:52,539 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:51:56,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:51:58,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:51:58,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:51:58,396 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:52:01,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:02,075 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:52:04,082 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:52:06,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:11,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:13,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:52:13,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:52:13,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:52:16,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:21,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:22,540 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:52:27,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:28,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:52:28,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:52:28,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:52:32,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:37,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:42,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:43,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:52:43,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:52:43,402 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:52:48,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:52,540 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:52:54,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:52:58,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:52:58,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:52:58,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:52:59,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:04,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:09,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:13,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:53:13,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:53:13,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:53:15,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:20,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:22,541 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:53:26,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:28,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:53:28,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:53:28,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:53:31,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:36,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:40,422 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 12:53:42,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:43,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:53:43,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:53:43,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:53:47,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:52,541 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:53:53,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:53:58,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:53:58,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:53:58,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:53:58,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:03,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:08,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:13,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:54:13,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:54:13,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:54:13,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:18,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:22,542 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:54:24,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:28,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:54:28,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:54:28,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:54:29,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:34,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:39,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:43,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:54:43,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:54:43,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:54:45,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:50,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:52,542 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:54:56,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:54:58,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:54:58,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:54:58,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:55:01,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:06,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:11,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:13,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:55:13,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:55:13,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:55:16,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:21,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:22,543 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:55:27,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:28,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:55:28,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:55:28,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:55:32,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:37,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:42,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:43,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:55:43,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:55:43,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:55:48,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:52,543 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:55:54,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:55:58,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:55:58,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:55:58,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:55:59,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:04,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:09,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:13,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:56:13,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:56:13,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:56:14,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:19,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:22,553 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:56:25,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:28,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:56:28,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:56:28,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:56:30,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:35,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:40,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:43,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:56:43,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:56:43,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:56:46,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:51,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:52,554 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:56:57,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:56:58,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:56:58,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:56:58,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:57:02,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:07,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:12,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:13,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:57:13,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:57:13,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:57:18,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:22,554 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:57:24,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:28,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:57:28,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:57:28,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:57:29,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:34,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:39,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:43,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:57:43,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:57:43,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:57:44,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:49,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:52,555 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:57:55,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:57:58,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:57:58,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:57:58,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:58:01,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:06,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:11,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:13,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:58:13,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:58:13,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:58:16,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:21,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:22,555 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:58:27,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:28,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:58:28,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:58:28,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:58:32,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:37,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:42,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:43,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:58:43,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:58:43,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:58:48,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:52,556 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:58:54,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:58:58,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:58:58,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:58:58,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:58:59,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:04,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:09,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:13,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:59:13,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:59:13,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:59:15,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:20,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:22,566 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:59:26,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:28,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:59:28,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:59:28,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:59:31,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:36,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:41,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:43,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:59:43,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:59:43,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 12:59:46,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:51,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:52,557 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 12:59:57,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 12:59:58,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 12:59:58,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 12:59:58,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:00:02,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:07,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:12,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:13,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:00:13,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:00:13,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:00:18,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:22,557 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:00:24,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:28,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:00:28,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:00:28,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:00:29,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:34,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:39,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:43,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:00:43,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:00:43,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:00:44,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:49,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:52,558 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:00:55,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:00:58,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:00:58,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:00:58,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:01:00,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:05,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:10,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:13,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:01:13,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:01:13,402 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:01:15,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:20,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:22,558 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:01:26,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:28,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:01:28,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:01:28,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:01:31,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:36,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:41,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:43,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:01:43,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:01:43,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:01:47,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:52,559 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:01:53,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:01:58,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:01:58,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:01:58,404 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:01:58,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:03,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:08,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:13,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:02:13,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:02:13,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:02:13,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:18,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:22,559 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:02:24,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:28,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:02:28,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:02:28,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:02:29,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:34,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:39,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:43,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:02:43,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:02:43,404 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:02:44,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:49,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:52,560 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:02:55,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:02:58,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:02:58,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:02:58,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:03:00,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:05,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:10,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:13,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:03:13,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:03:13,405 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:03:15,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:20,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:22,561 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:03:26,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:28,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:03:28,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:03:28,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:03:31,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:36,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:41,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:43,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:03:43,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:03:43,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:03:47,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:52,561 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:03:53,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:03:58,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:03:58,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:03:58,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:03:58,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:03,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:08,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:13,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:04:13,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:04:13,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:04:14,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:19,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:22,562 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:04:25,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:28,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:04:28,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:04:28,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:04:30,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:35,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:40,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:43,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:04:43,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:04:43,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:04:45,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:50,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:52,573 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:04:55,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:04:56,823 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:04:58,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:04:58,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:04:58,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:05:01,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:06,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:11,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:13,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:05:13,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:05:13,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:05:16,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:21,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:22,573 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:05:27,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:28,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:05:28,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:05:28,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:05:32,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:37,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:42,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:43,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:05:43,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:05:43,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:05:47,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:52,574 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:05:53,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:05:58,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:05:58,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:05:58,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:05:58,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:03,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:08,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:13,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:06:13,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:06:13,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:06:13,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:18,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:22,574 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:06:24,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:28,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:06:28,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:06:28,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:06:29,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:34,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:39,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:43,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:06:43,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:06:43,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:06:44,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:49,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:52,575 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:06:55,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:06:58,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:06:58,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:06:58,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:07:00,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:05,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:10,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:13,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:07:13,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:07:13,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:07:16,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:21,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:22,575 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:07:27,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:28,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:07:28,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:07:28,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:07:32,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:37,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:42,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:43,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:07:43,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:07:43,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:07:48,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:52,575 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:07:54,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:07:58,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:07:58,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:07:58,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:07:59,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:04,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:09,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:13,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:08:13,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:08:13,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:08:14,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:19,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:22,576 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:08:25,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:28,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:08:28,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:08:28,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:08:31,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:36,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:41,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:43,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:08:43,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:08:43,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:08:46,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:51,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:52,577 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:08:57,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:08:58,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:08:58,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:08:58,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:09:02,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:07,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:12,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:13,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:09:13,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:09:13,408 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:09:17,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:22,577 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:09:23,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:28,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:09:28,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:09:28,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:09:28,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:33,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:38,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:43,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:09:43,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:09:43,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:09:43,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:48,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:52,578 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:09:54,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:09:58,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:09:58,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:09:58,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:10:00,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:05,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:10,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:13,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:10:13,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:10:13,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:10:15,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:20,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:22,578 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:10:26,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:28,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:10:28,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:10:28,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:10:31,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:36,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:41,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:43,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:10:43,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:10:43,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:10:47,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:52,579 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:10:53,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:10:58,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:10:58,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:10:58,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:10:59,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:04,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:07,208 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:11:10,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:13,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:11:13,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:11:13,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:11:15,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:20,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:22,579 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:11:26,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:28,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:11:28,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:11:28,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:11:31,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:36,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:41,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:43,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:11:43,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:11:43,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:11:47,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:52,580 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:11:53,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:11:58,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:11:58,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:11:58,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:11:59,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:04,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:09,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:13,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:12:13,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:12:13,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:12:14,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:19,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:22,581 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:12:25,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:28,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:12:28,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:12:28,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:12:30,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:35,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:40,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:43,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:12:43,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:12:43,410 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:12:45,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:50,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:52,592 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:12:56,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:12:58,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:12:58,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:12:58,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:13:01,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:06,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:11,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:13,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:13:13,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:13:13,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:13:17,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:22,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:22,592 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:13:27,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:28,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:13:28,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:13:28,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:13:32,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:37,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:42,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:43,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:13:43,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:13:43,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:13:48,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:52,593 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:13:54,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:13:58,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:13:58,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:13:58,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:13:59,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:04,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:09,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:13,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:14:13,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:14:13,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:14:15,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:20,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:22,593 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:14:26,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:28,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:14:28,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:14:28,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:14:32,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:37,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:42,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:43,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:14:43,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:14:43,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:14:47,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:52,604 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:14:53,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:14:58,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:14:58,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:14:58,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:14:58,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:03,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:08,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:13,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:15:13,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:15:13,411 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:15:13,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:18,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:22,594 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:15:24,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:28,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:15:28,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:15:28,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:15:29,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:34,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:39,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:43,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:15:43,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:15:43,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:15:44,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:49,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:52,595 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:15:55,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:15:58,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:15:58,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:15:58,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:16:00,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:05,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:10,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:13,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:16:13,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:16:13,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:16:16,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:21,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:22,595 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:16:27,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:28,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:16:28,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:16:28,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:16:32,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:37,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:42,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:43,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:16:43,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:16:43,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:16:47,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:52,596 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:16:53,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:16:58,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:16:58,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:16:58,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:16:58,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:03,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:08,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:13,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:17:13,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:17:13,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:17:14,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:19,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:22,596 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:17:25,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:28,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:17:28,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:17:28,412 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:17:31,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:36,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:41,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:43,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:17:43,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:17:43,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:17:47,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:52,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:52,598 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:17:57,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:17:58,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:17:58,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:17:58,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:18:02,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:07,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:12,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:13,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:18:13,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:18:13,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:18:18,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:22,597 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:18:23,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:25,871 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:18:28,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:18:28,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:18:28,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:18:29,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:34,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:39,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:43,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:18:43,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:18:43,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:18:45,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:50,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:52,605 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:18:56,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:18:58,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:18:58,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:18:58,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:19:01,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:06,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:11,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:13,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:19:13,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:19:13,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:19:17,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:22,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:22,606 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:19:27,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:28,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:19:28,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:19:28,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:19:32,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:37,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:42,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:43,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:19:43,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:19:43,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:19:48,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:52,606 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:19:54,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:19:58,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:19:58,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:19:58,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:19:59,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:04,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:09,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:13,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:20:13,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:20:13,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:20:15,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:20,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:22,607 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:20:26,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:28,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:20:28,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:20:28,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:20:32,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:37,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:42,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:43,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:20:43,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:20:43,419 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:20:47,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:52,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:52,607 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:20:57,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:20:58,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:20:58,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:20:58,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:21:02,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:07,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:12,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:13,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:21:13,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:21:13,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:21:18,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:22,607 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:21:23,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:28,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:21:28,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:21:28,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:21:28,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:33,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:38,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:43,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:21:43,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:21:43,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:21:44,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:49,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:52,608 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:21:54,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:21:58,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:21:58,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:21:58,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:21:59,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:04,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:09,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:13,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:22:13,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:22:13,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:22:14,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:19,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:22,617 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:22:25,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:28,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:22:28,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:22:28,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:22:31,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:36,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:41,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:43,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:22:43,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:22:43,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:22:46,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:51,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:52,618 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:22:57,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:22:58,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:22:58,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:22:58,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:23:02,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:07,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:12,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:13,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:23:13,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:23:13,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:23:18,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:22,618 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:23:23,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:28,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:23:28,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:23:28,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:23:28,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:33,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:38,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:43,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:23:43,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:23:43,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:23:44,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:49,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:52,619 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:23:55,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:23:58,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:23:58,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:23:58,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:24:01,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:06,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:11,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:13,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:24:13,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:24:13,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:24:16,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:21,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:22,620 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:24:27,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:28,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:24:28,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:24:28,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:24:32,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:37,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:42,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:43,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:24:43,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:24:43,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:24:48,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:52,620 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:24:54,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:24:58,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:24:58,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:24:58,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:24:59,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:04,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:09,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:13,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:25:13,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:25:13,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:25:15,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:20,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:22,620 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:25:26,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:28,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:25:28,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:25:28,417 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:25:32,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:37,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:42,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:43,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:25:43,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:25:43,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:25:47,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:52,630 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:25:53,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:25:58,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:25:58,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:25:58,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:25:58,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:03,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:08,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:13,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:26:13,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:26:13,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:26:14,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:19,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:22,630 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:26:24,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:28,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:26:28,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:26:28,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:26:30,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:35,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:40,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:43,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:26:43,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:26:43,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:26:45,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:50,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:52,631 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:26:56,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:26:58,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:26:58,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:26:58,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:27:02,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:07,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:12,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:13,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:27:13,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:27:13,418 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:27:17,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:22,636 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:27:23,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:28,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:27:28,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:27:28,419 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:27:28,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:33,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:38,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:43,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:27:43,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:27:43,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:27:43,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:48,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:52,632 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:27:54,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:58,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:27:58,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:27:58,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:27:59,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:27:59,972 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:28:04,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:09,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:13,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:28:13,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:28:13,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:28:15,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:20,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:22,632 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:28:25,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:28,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:28:28,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:28:28,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:28:31,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:36,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:41,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:43,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:28:43,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:28:43,421 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:28:46,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:51,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:52,633 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:28:57,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:28:58,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:28:58,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:28:58,421 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:29:03,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:08,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:13,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:29:13,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:29:13,421 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:29:13,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:18,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:22,633 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:29:24,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:28,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:29:28,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:29:28,421 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:29:29,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:34,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:39,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:43,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:29:43,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:29:43,421 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:29:45,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:50,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:52,653 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:29:56,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:29:58,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:29:58,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:29:58,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:30:01,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:06,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:11,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:13,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:30:13,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:30:13,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:30:17,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:22,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:22,643 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:30:27,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:28,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:30:28,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:30:28,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:30:33,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:38,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:43,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:30:43,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:30:43,435 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:30:43,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:48,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:52,643 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:30:54,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:30:58,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:30:58,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:30:58,443 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:30:59,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:04,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:09,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:12,690 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:31:13,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:31:13,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:31:13,444 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:31:15,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:20,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:22,644 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:31:25,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:28,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:31:28,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:31:28,444 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:31:30,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:35,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:40,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:43,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:31:43,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:31:43,444 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:31:46,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:51,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:52,644 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:31:57,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:31:58,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:31:58,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:31:58,454 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:32:03,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:08,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:13,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:32:13,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:32:13,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:32:13,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:18,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:22,646 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:32:24,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:28,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:32:28,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:32:28,454 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:32:30,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:35,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:40,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:43,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:32:43,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:32:43,454 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:32:45,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:50,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:52,645 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:32:56,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:32:58,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:32:58,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:32:58,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:33:02,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:07,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:12,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:13,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:33:13,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:33:13,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:33:17,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:22,646 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:33:23,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:28,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:33:28,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:33:28,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:33:29,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:34,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:39,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:43,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:33:43,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:33:43,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:33:44,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:49,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:52,646 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:33:55,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:33:58,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:33:58,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:33:58,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:34:00,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:05,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:10,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:13,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:34:13,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:34:13,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:34:15,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:20,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:22,647 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:34:26,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:28,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:34:28,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:34:28,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:34:32,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:37,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:42,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:43,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:34:43,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:34:43,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:34:47,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:52,647 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:34:53,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:34:58,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:34:58,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:34:58,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:34:58,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:03,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:08,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:13,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:35:13,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:35:13,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:35:14,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:19,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:22,648 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:35:25,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:28,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:35:28,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:35:28,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:35:30,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:35,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:40,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:43,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:35:43,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:35:43,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:35:45,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:50,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:52,649 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:35:56,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:35:58,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:35:58,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:35:58,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:36:01,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:06,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:11,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:13,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:36:13,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:36:13,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:36:16,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:21,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:22,649 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:36:27,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:28,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:36:28,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:36:28,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:36:32,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:37,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:42,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:43,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:36:43,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:36:43,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:36:48,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:52,650 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:36:54,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:36:58,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:36:58,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:36:58,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:36:59,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:04,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:09,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:13,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:37:13,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:37:13,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:37:15,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:20,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:22,650 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:37:26,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:28,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:37:28,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:37:28,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:37:32,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:37,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:42,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:43,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:37:43,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:37:43,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:37:47,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:52,651 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:37:53,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:37:58,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:37:58,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:37:58,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:37:58,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:03,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:08,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:13,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:38:13,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:38:13,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:38:14,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:19,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:22,652 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:38:25,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:28,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:38:28,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:38:28,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:38:30,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:35,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:40,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:43,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:38:43,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:38:43,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:38:46,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:51,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:52,652 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:38:56,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:38:58,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:38:58,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:38:58,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:39:01,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:06,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:11,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:13,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:39:13,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:39:13,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:39:17,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:22,653 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:39:23,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:28,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:39:28,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:39:28,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:39:29,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:34,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:39,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:43,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:39:43,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:39:43,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:39:44,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:49,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:52,662 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:39:55,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:39:58,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:39:58,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:39:58,469 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:40:00,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:05,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:10,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:13,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:40:13,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:40:13,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:40:16,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:21,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:22,662 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:40:27,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:28,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:40:28,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:40:28,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:40:33,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:38,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:43,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:40:43,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:40:43,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:40:43,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:48,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:52,663 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:40:54,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:40:58,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:40:58,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:40:58,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:40:59,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:04,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:09,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:13,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:41:13,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:41:13,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:41:14,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:19,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:22,667 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:41:25,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:28,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:41:28,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:41:28,458 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:41:30,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:35,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:40,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:43,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:41:43,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:41:43,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:41:45,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:50,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:52,664 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:41:56,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:41:58,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:41:58,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:41:58,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:42:02,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:07,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:12,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:13,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:42:13,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:42:13,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:42:17,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:22,665 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:42:23,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:28,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:42:28,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:42:28,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:42:29,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:34,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:39,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:43,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:42:43,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:42:43,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:42:45,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:50,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:52,666 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:42:55,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:42:58,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:42:58,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:42:58,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:43:00,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:05,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:10,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:13,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:43:13,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:43:13,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:43:16,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:21,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:22,666 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:43:26,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:28,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:43:28,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:43:28,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:43:31,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:36,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:41,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:43,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:43:43,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:43:43,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:43:46,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:51,690 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:52,666 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:43:57,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:43:58,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:43:58,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:43:58,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:44:03,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:08,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:13,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:44:13,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:44:13,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:44:13,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:19,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:19,668 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:44:22,667 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:44:24,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:28,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:44:28,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:44:28,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:44:29,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:34,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:39,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:43,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:44:43,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:44:43,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:44:44,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:49,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:52,667 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:44:55,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:44:58,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:44:58,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:44:58,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:44:59,793 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:45:00,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:05,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:10,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:13,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:45:13,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:45:13,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:45:16,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:21,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:22,668 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:45:27,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:28,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:45:28,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:45:28,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:45:32,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:37,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:42,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:43,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:45:43,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:45:43,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:45:48,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:52,668 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:45:54,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:45:58,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:45:58,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:45:58,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:46:00,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:05,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:10,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:13,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:46:13,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:46:13,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:46:15,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:20,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:22,669 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:46:25,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:28,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:46:28,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:46:28,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:46:31,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:36,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:41,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:43,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:46:43,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:46:43,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:46:46,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:51,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:52,670 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:46:57,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:46:58,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:46:58,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:46:58,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:47:02,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:07,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:12,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:13,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:47:13,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:47:13,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:47:18,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:22,670 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:47:24,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:28,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:47:28,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:47:28,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:47:30,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:35,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:40,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:43,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:47:43,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:47:43,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:47:45,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:50,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:52,671 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:47:55,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:47:58,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:47:58,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:47:58,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:48:00,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:05,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:10,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:13,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:48:13,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:48:13,465 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:48:16,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:21,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:22,672 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:48:27,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:28,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:48:28,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:48:28,465 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:48:32,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:37,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:42,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:43,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:48:43,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:48:43,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:48:47,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:52,672 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:48:53,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:48:58,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:48:58,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:48:58,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:48:58,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:03,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:08,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:13,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:49:13,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:49:13,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:49:14,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:19,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:22,672 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:49:24,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:28,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:49:28,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:49:28,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:49:29,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:34,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:39,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:43,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:49:43,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:49:43,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:49:44,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:49,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:52,673 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:49:55,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:49:58,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:49:58,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:49:58,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:50:01,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:06,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:11,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:13,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:50:13,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:50:13,466 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:50:16,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:21,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:22,673 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:50:27,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:28,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:50:28,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:50:28,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:50:33,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:38,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:43,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:50:43,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:50:43,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:50:43,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:48,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:52,674 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:50:54,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:50:58,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:50:58,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:50:58,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:50:59,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:04,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:09,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:13,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:51:13,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:51:13,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:51:15,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:20,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:22,675 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:51:26,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:28,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:51:28,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:51:28,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:51:31,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:36,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:41,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:43,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:51:43,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:51:43,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:51:47,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:52,675 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:51:53,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:51:58,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:51:58,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:51:58,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:51:59,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:04,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:09,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:13,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:52:13,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:52:13,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:52:15,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:20,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:22,675 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:52:25,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:28,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:52:28,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:52:28,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:52:30,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:35,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:40,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:43,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:52:43,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:52:43,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:52:46,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:51,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:52,676 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:52:57,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:52:58,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:52:58,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:52:58,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:53:02,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:07,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:12,702 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:13,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:53:13,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:53:13,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:53:18,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:22,676 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:53:23,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:28,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:53:28,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:53:28,472 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:53:29,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:34,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:39,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:43,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:53:43,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:53:43,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:53:45,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:50,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:52,677 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:53:55,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:53:58,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:53:58,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:53:58,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:54:01,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:06,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:11,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:13,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:54:13,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:54:13,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:54:17,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:22,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:22,677 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:54:27,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:28,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:54:28,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:54:28,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:54:33,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:38,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:43,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:54:43,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:54:43,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:54:43,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:48,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:52,678 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:54:54,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:54:58,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:54:58,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:54:58,469 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:54:59,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:04,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:09,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:13,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:55:13,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:55:13,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:55:15,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:20,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:22,678 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:55:26,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:28,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:55:28,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:55:28,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:55:32,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:37,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:42,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:43,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:55:43,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:55:43,475 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:55:47,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:52,679 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:55:53,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:55:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:55:58,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:55:58,475 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:55:58,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:03,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:08,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:13,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:56:13,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:56:13,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:56:13,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:18,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:22,679 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:56:24,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:28,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:56:28,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:56:28,475 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:56:29,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:34,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:39,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:43,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:56:43,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:56:43,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:56:44,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:49,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:52,680 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:56:55,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:56:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:56:58,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:56:58,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:57:00,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:05,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:10,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:13,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:57:13,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:57:13,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:57:16,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:21,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:22,680 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:57:24,605 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 13:57:26,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:28,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:57:28,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:57:28,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:57:32,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:37,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:42,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:43,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:57:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:57:43,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:57:48,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:52,681 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:57:53,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:57:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:57:58,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:57:58,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:57:58,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:03,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:08,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:13,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:58:13,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:58:13,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:58:14,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:19,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:22,681 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:58:24,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:28,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:58:28,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:58:28,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:58:29,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:34,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:39,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:43,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:58:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:58:43,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:58:45,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:50,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:52,682 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:58:56,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:58:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:58:58,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:58:58,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:59:01,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:06,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:11,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:13,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:59:13,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:59:13,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:59:17,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:22,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:22,682 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:59:27,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:28,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:59:28,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:59:28,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:59:33,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:38,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:43,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:59:43,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:59:43,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:59:43,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:48,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:52,683 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 13:59:54,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 13:59:58,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 13:59:58,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 13:59:58,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 13:59:59,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:04,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:09,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:13,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:00:13,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:00:13,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:00:15,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:20,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:22,694 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:00:26,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:28,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:00:28,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:00:28,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:00:31,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:36,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:41,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:43,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:00:43,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:00:43,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:00:47,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:52,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:52,683 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:00:57,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:00:58,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:00:58,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:00:58,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:01:02,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:07,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:12,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:13,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:01:13,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:01:13,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:01:18,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:22,684 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:01:23,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:28,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:01:28,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:01:28,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:01:28,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:33,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:38,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:43,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:01:43,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:01:43,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:01:44,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:49,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:52,685 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:01:54,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:01:58,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:01:58,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:01:58,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:01:59,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:04,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:09,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:13,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:02:13,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:02:13,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:02:15,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:20,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:22,685 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:02:26,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:28,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:02:28,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:02:28,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:02:31,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:36,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:41,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:43,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:02:43,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:02:43,480 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:02:46,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:52,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:52,686 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:02:52,876 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:02:57,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:02:58,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:02:58,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:02:58,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:03:02,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:07,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:12,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:13,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:03:13,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:03:13,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:03:18,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:22,686 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:03:23,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:28,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:03:28,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:03:28,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:03:28,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:33,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:38,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:43,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:03:43,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:03:43,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:03:44,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:49,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:52,687 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:03:55,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:03:58,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:03:58,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:03:58,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:04:00,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:05,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:10,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:13,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:04:13,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:04:13,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:04:15,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:20,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:22,687 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:04:26,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:28,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:04:28,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:04:28,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:04:32,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:37,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:42,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:43,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:04:43,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:04:43,490 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:04:47,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:52,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:52,688 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:04:57,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:04:58,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:04:58,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:04:58,490 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:05:02,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:07,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:12,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:13,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:05:13,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:05:13,490 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:05:18,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:22,689 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:05:24,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:28,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:05:28,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:05:28,490 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:05:29,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:34,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:39,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:43,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:05:43,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:05:43,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:05:44,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:49,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:52,688 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:05:55,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:05:58,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:05:58,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:05:58,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:06:00,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:05,702 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:10,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:13,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:06:13,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:06:13,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:06:15,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:20,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:22,689 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:06:26,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:28,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:06:28,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:06:28,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:06:31,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:36,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:41,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:43,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:06:43,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:06:43,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:06:46,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:51,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:52,690 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:06:57,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:06:58,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:06:58,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:06:58,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:07:02,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:07,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:12,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:13,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:07:13,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:07:13,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:07:18,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:22,690 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:07:24,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:28,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:07:28,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:07:28,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:07:29,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:34,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:39,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:43,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:07:43,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:07:43,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:07:45,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:50,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:52,691 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:07:56,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:07:58,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:07:58,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:07:58,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:08:02,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:07,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:12,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:13,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:08:13,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:08:13,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:08:17,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:22,691 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:08:23,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:28,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:08:28,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:08:28,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:08:28,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:33,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:38,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:43,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:08:43,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:08:43,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:08:44,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:49,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:52,692 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:08:54,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:08:58,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:08:58,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:08:58,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:08:59,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:04,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:09,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:13,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:09:13,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:09:13,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:09:15,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:20,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:22,692 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:09:26,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:28,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:09:28,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:09:28,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:09:31,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:36,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:41,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:43,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:09:43,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:09:43,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:09:47,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:52,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:52,693 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:09:57,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:09:58,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:09:58,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:09:58,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:10:02,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:07,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:12,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:13,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:10:13,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:10:13,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:10:17,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:22,693 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:10:23,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:28,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:10:28,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:10:28,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:10:28,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:33,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:38,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:42,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:10:43,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:10:43,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:10:43,502 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:10:44,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:49,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:52,694 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:10:54,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:10:58,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:10:58,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:10:58,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:10:59,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:04,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:09,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:13,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:11:13,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:11:13,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:11:14,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:19,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:22,694 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:11:25,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:28,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:11:28,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:11:28,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:11:30,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:35,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:40,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:43,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:11:43,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:11:43,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:11:46,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:51,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:52,695 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:11:56,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:11:58,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:11:58,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:11:58,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:12:02,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:07,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:12,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:13,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:12:13,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:12:13,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:12:17,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:22,695 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:12:23,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:28,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:12:28,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:12:28,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:12:28,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:33,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:38,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:43,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:12:43,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:12:43,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:12:44,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:49,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:52,695 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:12:55,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:12:58,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:12:58,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:12:58,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:13:00,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:05,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:10,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:13,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:13:13,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:13:13,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:13:15,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:20,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:22,696 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:13:26,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:28,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:13:28,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:13:28,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:13:31,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:36,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:41,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:43,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:13:43,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:13:43,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:13:46,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:51,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:52,697 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:13:57,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:13:58,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:13:58,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:13:58,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:14:02,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:07,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:12,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:13,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:14:13,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:14:13,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:14:18,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:22,697 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:14:24,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:28,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:14:28,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:14:28,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:14:29,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:34,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:39,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:43,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:14:43,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:14:43,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:14:44,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:49,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:52,700 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:14:55,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:14:58,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:14:58,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:14:58,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:15:00,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:05,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:10,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:13,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:15:13,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:15:13,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:15:16,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:21,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:22,698 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:15:27,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:28,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:15:28,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:15:28,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:15:32,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:37,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:42,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:43,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:15:43,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:15:43,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:15:47,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:52,711 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:15:53,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:15:58,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:15:58,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:15:58,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:15:58,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:03,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:08,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:13,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:16:13,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:16:13,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:16:14,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:19,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:22,699 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:16:24,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:28,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:16:28,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:16:28,522 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:16:29,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:34,781 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:39,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:43,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:16:43,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:16:43,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:16:45,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:50,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:52,700 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:16:56,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:16:58,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:16:58,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:16:58,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:17:01,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:06,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:11,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:13,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:17:13,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:17:13,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:17:17,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:22,700 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:17:23,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:28,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:17:28,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:17:28,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:17:28,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:33,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:38,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:43,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:17:43,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:17:43,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:17:43,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:48,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:52,703 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:17:54,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:17:58,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:17:58,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:17:58,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:17:59,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:04,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:09,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:13,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:18:13,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:18:13,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:18:15,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:20,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:22,709 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:18:26,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:28,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:18:28,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:18:28,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:18:31,781 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:36,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:41,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:43,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:18:43,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:18:43,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:18:46,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:51,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:52,710 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:18:57,710 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:18:58,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:18:58,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:18:58,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:19:02,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:07,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:12,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:13,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:19:13,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:19:13,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:19:18,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:22,710 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:19:23,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:28,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:19:28,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:19:28,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:19:28,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:29,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:19:33,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:38,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:43,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:19:43,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:19:43,524 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:19:43,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:48,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:52,711 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:19:54,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:19:58,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:19:58,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:19:58,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:19:59,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:04,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:09,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:13,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:20:13,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:20:13,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:20:15,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:20,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:22,711 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:20:26,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:28,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:20:28,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:20:28,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:20:31,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:36,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:41,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:43,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:20:43,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:20:43,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:20:47,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:52,712 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:20:53,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:20:58,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:20:58,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:20:58,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:20:58,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:03,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:08,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:13,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:21:13,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:21:13,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:21:13,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:18,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:22,713 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:21:24,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:28,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:21:28,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:21:28,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:21:29,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:34,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:39,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:43,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:21:43,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:21:43,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:21:44,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:49,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:52,713 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:21:55,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:21:58,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:21:58,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:21:58,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:22:00,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:05,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:10,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:13,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:22:13,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:22:13,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:22:16,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:21,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:22,714 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:22:27,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:28,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:22:28,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:22:28,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:22:32,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:37,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:42,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:43,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:22:43,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:22:43,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:22:48,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:52,714 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:22:54,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:22:58,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:22:58,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:22:58,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:22:59,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:04,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:09,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:13,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:23:13,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:23:13,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:23:15,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:20,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:22,715 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:23:25,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:28,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:23:28,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:23:28,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:23:30,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:35,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:40,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:43,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:23:43,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:23:43,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:23:46,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:47,602 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:23:51,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:52,715 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:23:56,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:23:58,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:23:58,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:23:58,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:24:01,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:06,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:11,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:13,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:24:13,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:24:13,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:24:17,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:22,716 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:24:23,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:28,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:24:28,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:24:28,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:24:28,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:33,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:38,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:43,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:24:43,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:24:43,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:24:44,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:49,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:52,716 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:24:55,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:24:58,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:24:58,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:24:58,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:25:00,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:05,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:10,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:13,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:25:13,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:25:13,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:25:16,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:21,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:22,719 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:25:27,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:28,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:25:28,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:25:28,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:25:33,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:38,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:43,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:25:43,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:25:43,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:25:43,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:48,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:52,717 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:25:54,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:25:58,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:25:58,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:25:58,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:26:00,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:05,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:10,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:13,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:26:13,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:26:13,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:26:15,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:20,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:22,718 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:26:26,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:28,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:26:28,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:26:28,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:26:32,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:37,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:42,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:43,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:26:43,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:26:43,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:26:47,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:52,718 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:26:53,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:26:58,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:26:58,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:26:58,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:26:58,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:03,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:08,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:13,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:27:13,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:27:13,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:27:13,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:18,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:22,719 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:27:24,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:28,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:27:28,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:27:28,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:27:30,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:35,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:40,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:43,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:27:43,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:27:43,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:27:45,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:50,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:52,720 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:27:56,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:27:58,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:27:58,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:27:58,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:28:02,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:07,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:12,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:13,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:28:13,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:28:13,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:28:17,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:22,725 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:28:23,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:28,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:28:28,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:28:28,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:28:28,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:33,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:38,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:43,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:28:43,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:28:43,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:28:44,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:49,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:52,726 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:28:55,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:28:58,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:28:58,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:28:58,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:29:00,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:05,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:10,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:13,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:29:13,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:29:13,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:29:16,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:21,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:22,726 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:29:26,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:28,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:29:28,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:29:28,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:29:31,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:36,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:41,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:43,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:29:43,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:29:43,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:29:47,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:52,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:52,727 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:29:57,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:29:58,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:29:58,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:29:58,531 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:30:02,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:07,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:12,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:13,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:30:13,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:30:13,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:30:18,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:22,727 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:30:23,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:28,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:30:28,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:30:28,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:30:29,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:34,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:39,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:43,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:30:43,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:30:43,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:30:44,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:49,798 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:52,738 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:30:55,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:30:58,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:30:58,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:30:58,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:31:00,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:05,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:10,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:13,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:31:13,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:31:13,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:31:16,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:21,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:22,728 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:31:27,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:28,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:31:28,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:31:28,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:31:33,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:38,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:43,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:31:43,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:31:43,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:31:44,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:49,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:52,728 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:31:54,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:31:58,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:31:58,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:31:58,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:32:00,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:05,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:10,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:13,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:32:13,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:32:13,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:32:15,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:20,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:22,729 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:32:26,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:28,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:32:28,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:32:28,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:32:31,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:36,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:41,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:43,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:32:43,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:32:43,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:32:47,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:52,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:52,730 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:32:57,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:32:58,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:32:58,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:32:58,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:33:02,798 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:07,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:12,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:13,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:33:13,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:33:13,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:33:18,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:22,730 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:33:24,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:28,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:33:28,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:33:28,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:33:29,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:34,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:39,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:43,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:33:43,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:33:43,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:33:45,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:50,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:52,730 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:33:55,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:33:58,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:33:58,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:33:58,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:34:00,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:05,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:10,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:13,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:34:13,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:34:13,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:34:16,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:21,744 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:22,731 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:34:27,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:28,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:34:28,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:34:28,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:34:32,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:37,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:42,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:43,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:34:43,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:34:43,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:34:47,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:52,731 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:34:53,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:34:58,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:34:58,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:34:58,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:34:59,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:04,702 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:09,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:13,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:35:13,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:35:13,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:35:14,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:19,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:22,732 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:35:25,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:28,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:35:28,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:35:28,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:35:30,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:35,744 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:40,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:43,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:35:43,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:35:43,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:35:46,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:51,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:52,732 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:35:57,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:35:58,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:35:58,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:35:58,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:36:02,744 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:07,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:12,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:13,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:36:13,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:36:13,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:36:18,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:22,733 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:36:23,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:28,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:36:28,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:36:28,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:36:29,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:34,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:39,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:42,762 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:36:43,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:36:43,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:36:43,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:36:44,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:46,774 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:36:48,780 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:36:50,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:52,734 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:36:55,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:36:58,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:36:58,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:36:58,556 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:37:00,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:05,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:10,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:13,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:37:13,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:37:13,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:37:16,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:21,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:22,734 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:37:27,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:28,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:37:28,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:37:28,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:37:33,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:38,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:43,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:37:43,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:37:43,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:37:43,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:48,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:52,735 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:37:54,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:37:58,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:37:58,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:37:58,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:37:59,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:04,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:09,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:13,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:38:13,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:38:13,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:38:15,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:20,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:22,735 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:38:26,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:28,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:38:28,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:38:28,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:38:32,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:37,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:42,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:43,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:38:43,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:38:43,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:38:47,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:52,735 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:38:53,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:38:58,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:38:58,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:38:58,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:38:58,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:03,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:08,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:13,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:39:13,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:39:13,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:39:13,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:18,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:22,736 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:39:24,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:28,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:39:28,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:39:28,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:39:29,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:34,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:39,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:43,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:39:43,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:39:43,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:39:45,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:50,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:52,744 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:39:55,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:39:58,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:39:58,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:39:58,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:40:01,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:06,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:11,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:13,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:40:13,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:40:13,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:40:16,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:21,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:22,745 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:40:27,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:28,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:40:28,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:40:28,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:40:33,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:38,744 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:40:43,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:40:43,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:40:43,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:48,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:52,745 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:40:54,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:40:58,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:40:58,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:40:58,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:40:59,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:04,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:09,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:13,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:41:13,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:41:13,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:41:15,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:20,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:22,745 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:41:26,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:28,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:41:28,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:41:28,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:41:31,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:36,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:41,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:41:43,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:41:43,545 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:41:46,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:51,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:52,746 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:41:57,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:41:58,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:41:58,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:41:58,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:42:02,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:07,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:12,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:13,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:42:13,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:42:13,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:42:18,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:22,750 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:42:23,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:28,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:42:28,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:42:28,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:42:28,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:33,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:38,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:43,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:42:43,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:42:43,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:42:43,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:48,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:52,748 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:42:54,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:42:58,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:42:58,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:42:58,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:42:59,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:04,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:09,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:13,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:43:13,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:43:13,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:43:15,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:20,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:22,749 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:43:26,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:28,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:43:28,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:43:28,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:43:31,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:36,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:41,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:43,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:43:43,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:43:43,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:43:47,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:52,749 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:43:53,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:43:58,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:43:58,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:43:58,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:43:59,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:04,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:09,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:13,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:44:13,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:44:13,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:44:14,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:19,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:22,750 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:44:25,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:28,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:44:28,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:44:28,539 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:44:30,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:35,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:40,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:43,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:44:43,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:44:43,539 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:44:46,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:51,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:52,754 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:44:56,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:44:58,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:44:58,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:44:58,539 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:45:02,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:07,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:12,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:13,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:45:13,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:45:13,539 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:45:17,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:22,752 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:45:23,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:28,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:45:28,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:45:28,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:45:28,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:33,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:38,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:43,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:45:43,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:45:43,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:45:44,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:49,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:52,754 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:45:54,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:45:58,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:45:58,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:45:58,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:46:00,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:05,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:10,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:13,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:46:13,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:46:13,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:46:16,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:21,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:22,753 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:46:26,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:28,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:46:28,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:46:28,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:46:32,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:37,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:42,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:43,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:46:43,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:46:43,541 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:46:47,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:52,754 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:46:53,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:46:58,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:46:58,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:46:58,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:46:58,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:03,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:08,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:13,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:47:13,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:47:13,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:47:14,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:19,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:22,754 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:47:25,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:28,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:47:28,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:47:28,541 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:47:30,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:35,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:40,796 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:43,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:47:43,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:47:43,541 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:47:46,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:51,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:52,755 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:47:57,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:47:58,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:47:58,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:47:58,542 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:48:02,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:07,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:12,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:13,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:48:13,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:48:13,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:48:17,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:22,756 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:48:23,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:28,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:48:28,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:48:28,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:48:28,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:33,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:38,983 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:43,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:48:43,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:48:43,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:48:44,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:49,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:52,756 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:48:55,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:48:58,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:48:58,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:48:58,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:49:00,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:05,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:10,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:13,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:49:13,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:49:13,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:49:16,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:21,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:22,757 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:49:27,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:28,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:49:28,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:49:28,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:49:32,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:37,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:42,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:43,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:49:43,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:49:43,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:49:48,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:52,757 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:49:53,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:49:58,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:49:58,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:49:58,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:49:59,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:04,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:09,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:13,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:50:13,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:50:13,785 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:50:14,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:19,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:22,758 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:50:24,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:28,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:50:28,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:50:28,785 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:50:29,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:33,984 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:50:35,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:40,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:43,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:50:43,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:50:43,785 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:50:46,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:51,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:52,759 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:50:56,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:50:58,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:50:58,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:50:58,786 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:51:02,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:07,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:12,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:13,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:51:13,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:51:13,786 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:51:18,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:22,759 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:51:23,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:28,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:51:28,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:51:28,785 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:51:28,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:33,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:38,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:43,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:51:43,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:51:43,785 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:51:44,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:49,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:52,760 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:51:54,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:51:58,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:51:58,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:51:58,786 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:52:00,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:05,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:10,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:13,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:52:13,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:52:13,794 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:52:15,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:20,082 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:22,761 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:52:25,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:28,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:52:28,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:52:28,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:52:31,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:36,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:41,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:43,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:52:43,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:52:43,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:52:46,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:51,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:52,761 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:52:56,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:52:58,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:52:58,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:52:58,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:53:02,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:07,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:12,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:13,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:53:13,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:53:13,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:53:17,984 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:22,762 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:53:23,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:28,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:53:28,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:53:28,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:53:28,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:33,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:38,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:43,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:53:43,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:53:43,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:53:44,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:49,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:52,762 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:53:54,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:53:58,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:53:58,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:53:58,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:54:00,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:05,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:10,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:13,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:54:13,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:54:13,796 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:54:15,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:20,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:22,763 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:54:26,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:28,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:54:28,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:54:28,796 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:54:32,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:37,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:42,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:43,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:54:43,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:54:43,795 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:54:48,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:52,763 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:54:53,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:58,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:54:58,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:54:58,802 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:54:59,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:54:59,990 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 14:55:04,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:09,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:13,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:55:13,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:55:13,802 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:55:14,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:19,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:22,764 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:55:24,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:28,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:55:28,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:55:28,803 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:55:29,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:34,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:39,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:43,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:55:43,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:55:43,803 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:55:45,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:50,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:52,764 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:55:55,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:55:58,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:55:58,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:55:58,803 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:56:01,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:06,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:11,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:13,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:56:13,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:56:13,805 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:56:17,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:22,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:22,765 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:56:27,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:28,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:56:28,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:56:28,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:56:32,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:37,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:42,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:43,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:56:43,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:56:43,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:56:48,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:52,765 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:56:53,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:56:58,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:56:58,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:56:58,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:56:59,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:04,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:09,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:13,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:57:13,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:57:13,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:57:15,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:20,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:22,766 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:57:25,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:28,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:57:28,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:57:28,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:57:31,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:36,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:41,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:43,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:57:43,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:57:43,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:57:46,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:51,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:52,766 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:57:57,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:57:58,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:57:58,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:57:58,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:58:03,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:08,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:13,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:13,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:58:13,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:58:13,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:58:18,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:22,767 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:58:23,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:28,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:58:28,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:58:28,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:58:29,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:34,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:39,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:43,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:58:43,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:58:43,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:58:44,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:49,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:52,767 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:58:54,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:58:58,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:58:58,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:58:58,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:59:00,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:05,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:10,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:13,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:59:13,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:59:13,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:59:15,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:20,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:22,768 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:59:26,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:28,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:59:28,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:59:28,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:59:32,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:37,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:42,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:43,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:59:43,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:59:43,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:59:47,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:52,769 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 14:59:53,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 14:59:58,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 14:59:58,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 14:59:58,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 14:59:59,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:04,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:09,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:13,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:00:13,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:00:13,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:00:14,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:19,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:22,769 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:00:25,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:28,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:00:28,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:00:28,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:00:31,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:36,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:41,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:00:43,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:00:43,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:00:46,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:51,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:52,770 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:00:56,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:00:58,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:00:58,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:00:58,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:01:01,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:06,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:11,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:13,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:01:13,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:01:13,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:01:17,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:22,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:22,781 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:01:27,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:28,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:01:28,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:01:28,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:01:33,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:38,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:43,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:01:43,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:01:43,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:01:49,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:52,771 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:01:54,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:01:58,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:01:58,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:01:58,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:02:00,092 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:05,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:10,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:13,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:02:13,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:02:13,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:02:15,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:20,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:22,771 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:02:25,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:28,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:02:28,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:02:28,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:02:30,983 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:35,984 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:40,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:02:43,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:02:43,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:02:46,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:51,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:52,775 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:02:56,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:02:58,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:02:58,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:02:58,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:03:02,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:07,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:12,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:13,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:03:13,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:03:13,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:03:18,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:22,773 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:03:23,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:28,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:03:28,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:03:28,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:03:28,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:33,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:39,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:43,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:03:43,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:03:43,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:03:44,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:47,053 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:03:49,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:52,772 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:03:54,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:03:58,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:03:58,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:03:58,818 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:04:00,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:05,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:10,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:13,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:04:13,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:04:13,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:04:15,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:20,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:22,773 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:04:25,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:28,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:04:28,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:04:28,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:04:30,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:35,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:40,983 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:43,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:04:43,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:04:43,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:04:46,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:51,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:52,774 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:04:56,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:04:58,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:04:58,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:04:58,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:05:02,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:07,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:12,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:13,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:05:13,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:05:13,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:05:17,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:22,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:22,778 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:05:27,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:28,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:05:28,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:05:28,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:05:33,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:38,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:43,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:43,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:05:43,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:05:43,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:05:49,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:52,779 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:05:54,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:05:58,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:05:58,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:05:58,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:06:00,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:05,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:10,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:13,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:06:13,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:06:13,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:06:16,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:21,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:22,779 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:06:26,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:28,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:06:28,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:06:28,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:06:31,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:36,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:41,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:43,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:06:43,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:06:43,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:06:47,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:52,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:52,780 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:06:57,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:06:58,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:06:58,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:06:58,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:07:03,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:08,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:13,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:13,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:07:13,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:07:13,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:07:18,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:22,781 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:07:24,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:28,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:07:28,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:07:28,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:07:30,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:35,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:40,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:43,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:07:43,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:07:43,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:07:45,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:50,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:52,781 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:07:55,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:07:58,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:07:58,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:07:58,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:08:00,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:05,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:10,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:13,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:08:13,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:08:13,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:08:16,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:21,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:22,782 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:08:26,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:28,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:08:28,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:08:28,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:08:32,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:37,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:42,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:43,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:08:43,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:08:43,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:08:47,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:52,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:52,782 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:08:57,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:08:58,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:08:58,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:08:58,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:09:02,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:07,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:12,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:13,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:09:13,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:09:13,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:09:18,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:22,783 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:09:23,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:28,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:09:28,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:09:28,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:09:29,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:34,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:39,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:43,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:09:43,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:09:43,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:09:44,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:49,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:52,783 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:09:54,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:09:58,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:09:58,821 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:09:58,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:10:00,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:05,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:10,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:13,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:10:13,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:10:13,826 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:10:15,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:20,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:22,784 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:10:25,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:28,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:10:28,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:10:28,826 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:10:31,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:36,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:41,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:43,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:10:43,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:10:43,826 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:10:46,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:51,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:52,784 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:10:56,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:10:58,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:10:58,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:10:58,826 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:11:02,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:07,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:12,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:13,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:11:13,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:11:13,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:11:18,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:22,785 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:11:23,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:28,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:11:28,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:11:28,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:11:29,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:34,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:39,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:43,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:11:43,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:11:43,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:11:45,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:50,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:52,787 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:11:55,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:11:58,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:11:58,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:11:58,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:12:01,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:06,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:11,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:13,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:12:13,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:12:13,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:12:16,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:21,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:22,787 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:12:26,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:28,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:12:28,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:12:28,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:12:32,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:35,034 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:12:37,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:42,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:43,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:12:43,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:12:43,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:12:47,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:52,788 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:12:53,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:12:58,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:12:58,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:12:58,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:12:59,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:04,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:09,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:13,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:13:13,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:13:13,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:13:15,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:20,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:22,788 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:13:25,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:28,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:13:28,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:13:28,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:13:31,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:36,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:41,014 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:43,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:13:43,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:13:43,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:13:46,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:51,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:52,789 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:13:56,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:13:58,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:13:58,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:13:58,827 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:14:02,086 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:07,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:12,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:13,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:14:13,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:14:13,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:14:18,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:22,790 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:14:23,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:28,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:14:28,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:14:28,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:14:29,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:34,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:39,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:43,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:14:43,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:14:43,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:14:44,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:49,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:52,790 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:14:54,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:14:58,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:14:58,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:14:58,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:15:00,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:05,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:10,004 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:13,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:15:13,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:15:13,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:15:15,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:20,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:22,790 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:15:25,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:28,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:15:28,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:15:28,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:15:31,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:36,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:41,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:43,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:15:43,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:15:43,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:15:46,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:51,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:52,791 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:15:56,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:15:58,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:15:58,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:15:58,828 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:16:02,038 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:07,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:12,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:13,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:16:13,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:16:13,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:16:17,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:22,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:22,803 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:16:27,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:28,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:16:28,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:16:28,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:16:32,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:37,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:42,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:43,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:16:43,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:16:43,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:16:48,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:52,792 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:16:53,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:16:58,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:16:58,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:16:58,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:16:59,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:04,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:09,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:13,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:17:13,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:17:13,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:17:15,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:20,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:22,792 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:17:25,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:28,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:17:28,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:17:28,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:17:31,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:35,092 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:17:36,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:41,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:43,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:17:43,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:17:43,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:17:47,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:52,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:52,793 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:17:57,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:17:58,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:17:58,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:17:58,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:18:02,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:07,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:12,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:13,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:18:13,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:18:13,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:18:18,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:22,794 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:18:23,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:28,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:18:28,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:18:28,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:18:29,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:34,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:39,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:43,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:18:43,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:18:43,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:18:45,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:50,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:52,794 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:18:55,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:18:58,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:18:58,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:18:58,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:19:01,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:06,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:11,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:13,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:19:13,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:19:13,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:19:16,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:21,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:22,795 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:19:27,796 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:28,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:19:28,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:19:28,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:19:33,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:38,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:43,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:43,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:19:43,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:19:43,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:19:49,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:52,796 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:19:54,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:19:58,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:19:58,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:19:58,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:20:00,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:05,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:10,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:13,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:20:13,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:20:13,831 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:20:15,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:20,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:22,796 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:20:25,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:28,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:20:28,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:20:28,831 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:20:31,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:36,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:41,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:43,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:20:43,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:20:43,835 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:20:46,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:51,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:52,805 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:20:56,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:20:58,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:20:58,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:20:58,846 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:21:02,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:07,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:12,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:13,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:21:13,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:21:13,846 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:21:18,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:22,806 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:21:23,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:28,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:21:28,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:21:28,846 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:21:29,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:34,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:39,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:43,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:21:43,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:21:43,846 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:21:44,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:49,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:52,806 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:21:54,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:21:58,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:21:58,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:21:58,847 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:22:00,113 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:05,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:10,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:13,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:22:13,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:22:13,847 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:22:16,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:21,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:22,806 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:22:26,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:28,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:22:28,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:22:28,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:22:32,083 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:37,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:42,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:43,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:22:43,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:22:43,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:22:47,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:52,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:52,807 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:22:57,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:22:58,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:22:58,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:22:58,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:23:03,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:08,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:13,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:13,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:23:13,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:23:13,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:23:18,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:22,807 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:23:23,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:28,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:23:28,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:23:28,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:23:29,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:34,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:39,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:43,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:23:43,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:23:43,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:23:44,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:49,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:52,808 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:23:54,809 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:23:58,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:23:58,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:23:58,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:24:00,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:05,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:10,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:13,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:24:13,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:24:13,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:24:16,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:21,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:22,816 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:24:26,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:28,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:24:28,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:24:28,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:24:32,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:37,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:42,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:43,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:24:43,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:24:43,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:24:47,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:52,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:52,816 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:24:57,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:24:58,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:24:58,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:24:58,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:25:03,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:08,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:13,113 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:13,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:25:13,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:25:13,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:25:19,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:22,816 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:25:24,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:28,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:25:28,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:25:28,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:25:30,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:35,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:40,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:43,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:25:43,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:25:43,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:25:45,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:50,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:52,817 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:25:55,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:25:58,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:25:58,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:25:58,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:26:01,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:06,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:11,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:13,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:26:13,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:26:13,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:26:17,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:22,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:22,817 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:26:27,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:28,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:26:28,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:26:28,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:26:33,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:38,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:43,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:43,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:26:43,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:26:43,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:26:48,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:52,818 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:26:53,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:26:58,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:26:58,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:26:58,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:26:59,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:04,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:09,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:13,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:27:13,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:27:13,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:27:14,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:19,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:22,819 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:27:24,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:28,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:27:28,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:27:28,857 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:27:30,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:35,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:40,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:43,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:27:43,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:27:43,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:27:45,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:50,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:52,819 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:27:55,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:27:58,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:27:58,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:27:58,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:28:01,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:06,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:11,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:13,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:28:13,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:28:13,857 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:28:16,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:21,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:22,820 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:28:26,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:28,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:28:28,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:28:28,857 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:28:32,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:37,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:42,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:43,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:28:43,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:28:43,858 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:28:47,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:52,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:52,820 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:28:57,821 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:28:58,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:28:58,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:28:58,858 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:29:03,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:08,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:13,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:13,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:29:13,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:29:13,858 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:29:18,113 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:22,658 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:29:22,821 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:29:23,821 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:24,664 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:29:28,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:29:28,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:29:28,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:29:29,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:34,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:39,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:43,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:29:43,858 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:29:43,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:29:44,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:49,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:52,821 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:29:54,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:29:58,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:29:58,858 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:29:58,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:30:00,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:05,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:10,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:13,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:30:13,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:30:13,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:30:15,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:22,822 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:30:24,877 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:30:25,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:28,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:30:28,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:30:28,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:30:31,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:36,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:41,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:43,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:30:43,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:30:43,859 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:30:46,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:51,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:52,833 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:30:56,834 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:30:58,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:30:58,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:30:58,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:31:02,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:07,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:12,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:13,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:31:13,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:31:13,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:31:18,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:22,823 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:31:23,823 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:28,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:31:28,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:31:28,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:31:29,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:34,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:39,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:43,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:31:43,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:31:43,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:31:45,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:50,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:52,824 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:31:55,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:31:58,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:31:58,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:31:58,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:32:01,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:06,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:11,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:13,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:32:13,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:32:13,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:32:17,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:22,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:22,824 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:32:27,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:28,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:32:28,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:32:28,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:32:33,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:38,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:43,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:43,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:32:43,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:32:43,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:32:49,092 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:52,825 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:32:54,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:32:58,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:32:58,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:32:58,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:33:00,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:05,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:10,109 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:13,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:33:13,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:33:13,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:33:16,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:21,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:22,825 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:33:26,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:28,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:33:28,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:33:28,861 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:33:32,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:37,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:42,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:43,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:33:43,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:33:43,861 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:33:47,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:52,092 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:52,826 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:33:57,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:33:58,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:33:58,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:33:58,861 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:34:03,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:08,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:13,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:13,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:34:13,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:34:13,861 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:34:19,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:22,826 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:34:24,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:28,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:34:28,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:34:28,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:34:30,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:35,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:40,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:43,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:34:43,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:34:43,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:34:46,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:51,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:52,826 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:34:56,842 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:34:58,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:34:58,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:34:58,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:35:02,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:07,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:12,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:13,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:35:13,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:35:13,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:35:17,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:22,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:22,827 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:35:27,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:28,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:35:28,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:35:28,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:35:33,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:38,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:43,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:43,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:35:43,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:35:43,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:35:49,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:52,827 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:35:54,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:35:58,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:35:58,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:35:58,867 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:36:00,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:05,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:10,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:13,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:36:13,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:36:13,867 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:36:15,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:20,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:22,828 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:36:25,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:28,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:36:28,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:36:28,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:36:31,094 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:36,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:41,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:43,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:36:43,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:36:43,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:36:46,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:51,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:52,828 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:36:56,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:36:58,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:36:58,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:36:58,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:37:02,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:07,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:12,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:13,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:37:13,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:37:13,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:37:17,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:22,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:22,829 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:37:27,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:28,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:37:28,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:37:28,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:37:33,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:38,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:43,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:43,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:37:43,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:37:43,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:37:49,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:52,829 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:37:54,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:37:58,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:37:58,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:37:58,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:38:00,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:05,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:10,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:13,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:38:13,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:38:13,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:38:15,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:20,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:22,830 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:38:25,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:28,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:38:28,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:38:28,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:38:31,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:36,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:41,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:43,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:38:43,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:38:43,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:38:47,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:52,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:52,830 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:38:57,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:38:58,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:38:58,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:38:58,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:39:03,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:08,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:13,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:13,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:39:13,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:39:13,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:39:19,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:22,831 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:39:24,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:28,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:39:28,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:39:28,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:39:30,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:35,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:40,082 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:43,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:39:43,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:39:43,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:39:45,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:50,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:52,831 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:39:55,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:39:58,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:39:58,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:39:58,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:40:01,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:06,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:11,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:13,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:40:13,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:40:13,890 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:40:17,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:22,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:22,832 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:40:27,833 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:28,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:40:28,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:40:28,889 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:40:33,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:38,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:43,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:43,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:40:43,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:40:43,890 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:40:48,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:52,833 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:40:53,833 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:40:58,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:40:58,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:40:58,890 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:40:59,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:04,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:09,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:13,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:41:13,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:41:13,890 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:41:15,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:20,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:22,845 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:41:25,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:28,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:41:28,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:41:28,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:41:31,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:36,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:41,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:43,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:41:43,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:41:43,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:41:47,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:52,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:52,846 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:41:57,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:41:58,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:41:58,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:41:58,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:42:03,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:08,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:13,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:13,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:42:13,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:42:13,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:42:18,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:22,847 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:42:23,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:28,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:42:28,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:42:28,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:42:29,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:34,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:39,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:43,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:42:43,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:42:43,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:42:44,108 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:49,109 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:52,847 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:42:54,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:42:58,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:42:58,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:42:58,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:43:00,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:05,082 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:10,083 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:13,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:43:13,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:43:13,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:43:16,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:21,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:22,848 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:43:26,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:28,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:43:28,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:43:28,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:43:32,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:37,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:42,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:43,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:43:43,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:43:43,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:43:47,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:52,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:52,848 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:43:52,918 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:43:57,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:43:58,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:43:58,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:43:58,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:44:03,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:08,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:13,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:13,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:44:13,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:44:13,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:44:18,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:22,859 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:44:23,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:28,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:44:28,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:44:28,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:44:29,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:34,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:39,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:43,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:44:43,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:44:43,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:44:44,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:49,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:52,859 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:44:54,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:44:58,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:44:58,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:44:58,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:45:00,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:05,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:10,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:13,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:45:13,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:45:13,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:45:16,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:21,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:22,860 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:45:26,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:28,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:45:28,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:45:28,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:45:32,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:37,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:42,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:43,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:45:43,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:45:43,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:45:47,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:52,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:52,860 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:45:57,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:45:58,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:45:58,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:45:58,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:46:03,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:08,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:13,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:13,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:46:13,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:46:13,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:46:19,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:22,861 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:46:24,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:28,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:46:28,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:46:28,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:46:30,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:35,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:40,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:43,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:46:43,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:46:43,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:46:45,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:47,531 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:46:49,539 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:46:50,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:52,861 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:46:55,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:46:58,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:46:58,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:46:58,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:47:01,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:06,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:11,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:13,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:47:13,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:47:13,894 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:47:16,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:21,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:22,862 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:47:26,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:28,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:47:28,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:47:28,894 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:47:29,715 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:47:32,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:37,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:42,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:43,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:47:43,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:47:43,894 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:47:48,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:52,862 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:47:53,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:47:58,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:03,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:08,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:13,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:18,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:22,873 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:48:23,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:28,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:33,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:38,901 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:43,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:48,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:52,865 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:48:54,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:48:59,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:04,868 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:09,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:14,870 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:15,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:49:15,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:49:15,943 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:49:20,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:22,881 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:49:25,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:30,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:49:30,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:49:30,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:49:31,092 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:36,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:37,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:49:41,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:45,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:49:45,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:49:45,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:49:47,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:52,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:49:52,874 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:49:57,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:02,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:07,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:12,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:17,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:22,875 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:50:23,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:28,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:32,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:50:32,614 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:50:32,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:50:34,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:35,679 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:50:39,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:44,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:47,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:50:47,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:50:47,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:50:47,880 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:50:49,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:52,875 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:50:54,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:50:57,932 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:50:59,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:02,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:51:02,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:51:02,614 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:51:05,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:10,003 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:51:11,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:16,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:17,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:51:17,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:51:17,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:51:20,064 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:51:21,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:22,876 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:51:27,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:30,115 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:51:32,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:51:32,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:51:32,615 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:51:32,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:38,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:40,170 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:51:43,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:47,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:51:47,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:51:47,615 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:51:48,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:50,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:51:52,876 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:51:53,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:51:59,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:00,310 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:02,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:52:02,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:52:02,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:52:04,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:09,369 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:10,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:15,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:17,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:52:17,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:52:17,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:52:19,429 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:20,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:22,877 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:52:26,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:29,483 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:32,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:52:32,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:32,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:52:32,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:52:38,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:39,541 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:43,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:47,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:52:47,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:52:47,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:52:48,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:49,600 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:52:52,877 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:52:53,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:59,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:52:59,659 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:02,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:53:02,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:53:02,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:53:04,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:09,725 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:10,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:15,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:17,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:53:17,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:53:17,618 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:53:19,779 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:20,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:22,878 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:53:25,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:29,835 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:31,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:32,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:53:32,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:53:32,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:53:36,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:38,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 15:53:38,224 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 15:53:38,229 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 15:53:38,230 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 15:53:38,231 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 15:53:38,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 15:53:38,246 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 15:53:38,893 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 15:53:39,900 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:40,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:42,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:42,912 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/config.yaml 2023-11-22 15:53:47,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:47,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:53:47,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:53:47,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:53:49,973 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:53:52,879 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:53:52,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:53:57,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:00,042 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:02,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:54:02,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:54:02,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:54:03,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:09,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:10,111 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:14,590 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:17,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:54:17,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:54:17,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:54:19,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:20,175 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:22,879 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:54:25,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:30,236 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:31,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:32,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:54:32,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:54:32,616 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:54:36,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:40,327 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:41,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:46,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:47,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:54:47,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:54:47,640 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:54:50,391 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:54:52,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:54:52,880 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:54:57,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:00,446 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:02,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:55:02,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:55:02,638 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:55:03,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:08,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:10,520 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:14,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:18,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:55:18,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:55:18,019 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:55:19,579 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:20,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:22,880 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:55:25,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:29,647 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:31,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:33,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:55:33,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:55:33,016 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:55:37,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:39,713 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:43,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:48,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:48,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:55:48,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:55:48,446 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:55:51,787 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:55:52,881 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:55:53,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:55:58,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:01,936 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:03,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:56:03,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:56:03,444 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:56:04,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:09,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:12,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:15,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:19,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:56:19,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:56:19,025 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:56:20,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:22,072 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:22,892 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:56:25,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:31,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:32,133 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:34,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:56:34,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:56:34,025 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:56:37,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:42,207 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:42,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:47,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:49,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:56:49,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:56:49,551 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:56:52,271 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:56:52,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:56:52,882 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:56:57,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:01,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 15:57:01,068 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 15:57:01,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 15:57:01,071 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 15:57:01,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 15:57:02,358 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:03,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:03,365 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:04,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:57:04,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:57:04,548 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:57:08,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:12,431 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:14,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:19,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:19,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:57:19,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:57:19,880 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:57:22,509 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:22,883 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:57:24,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:29,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:34,652 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:34,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:57:34,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:57:34,875 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:57:35,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:40,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:44,717 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:45,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:50,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:57:50,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:57:50,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:57:50,723 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:57:52,883 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:57:53,782 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:57:55,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:00,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:03,881 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:05,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:58:05,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:58:05,721 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:58:05,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:10,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:13,950 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:16,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:21,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:21,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:58:21,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:58:21,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:58:22,884 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:58:24,013 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:26,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:31,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:34,100 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:36,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:58:36,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:58:36,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:58:37,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:43,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:44,163 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:48,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:51,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:58:51,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:58:51,444 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:58:52,884 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:58:53,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:58:54,231 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:58:58,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:04,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:04,303 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:06,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:59:06,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:59:06,442 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:59:09,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:14,370 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:14,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:19,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:21,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:59:21,781 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:59:21,782 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:59:22,886 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:59:24,429 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:24,911 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:29,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:34,491 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:35,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:36,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:59:36,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:59:36,781 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:59:40,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:44,564 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:46,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:51,781 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 15:59:51,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 15:59:51,784 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 15:59:52,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 15:59:52,886 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 15:59:54,635 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 15:59:57,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:02,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:04,698 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:06,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:00:06,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:00:06,781 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:00:08,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:13,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:14,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:18,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:22,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:00:22,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:00:22,464 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:00:22,887 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:00:23,836 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:23,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:00:23,887 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:00:23,887 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:00:23,889 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:00:24,836 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:24,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:00:26,844 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:28,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:33,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:36,909 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:37,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:00:37,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:00:37,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:00:39,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:45,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:45,980 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:00:50,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:52,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:00:52,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:00:52,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:00:52,888 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:00:55,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:00:58,071 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:01,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:06,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:07,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:01:07,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:01:07,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:01:08,174 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:12,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:18,242 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:18,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:22,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:01:22,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:01:22,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:01:22,888 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:01:23,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:28,303 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:29,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:34,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:37,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:01:37,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:01:37,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:01:38,463 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:39,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:44,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:48,527 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:01:50,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:52,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:01:52,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:01:52,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:01:52,889 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:01:55,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:01:58,593 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:01,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:06,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:07,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:02:07,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:02:07,463 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:02:08,674 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:11,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:16,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:18,743 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:22,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:22,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:02:22,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:02:22,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:02:22,889 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:02:28,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:28,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:33,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:37,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:02:37,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:02:37,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:02:38,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:38,915 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:43,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:48,989 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:02:49,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:52,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:02:52,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:02:52,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:02:52,890 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:02:54,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:02:59,054 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:00,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:05,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:07,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:03:07,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:03:07,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:03:09,118 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:10,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:15,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:19,177 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:21,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:22,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:03:22,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:03:22,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:03:22,890 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:03:26,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:28,254 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:32,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:37,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:37,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:03:37,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:03:37,463 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:03:38,344 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:42,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:47,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:52,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:03:52,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:03:52,463 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:03:52,891 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:03:52,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:03:53,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:03:53,263 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:03:53,264 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:03:53,266 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:03:53,438 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:03:54,443 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:55,450 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:03:58,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:03,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:04,518 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:07,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:04:07,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:04:07,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:04:08,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:14,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:14,592 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:19,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:22,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:04:22,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:04:22,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:04:22,891 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:04:24,661 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:25,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:30,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:34,730 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:35,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:37,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:04:37,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:04:37,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:04:41,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:44,805 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:47,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:52,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:04:52,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:04:52,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:04:52,482 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:04:52,893 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:04:54,878 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:04:57,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:03,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:04,948 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:07,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:05:07,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:05:07,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:05:09,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:15,019 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:15,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:20,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:22,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:05:22,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:05:22,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:05:22,893 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:05:25,089 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:25,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:30,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:35,157 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:36,202 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:37,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:05:37,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:05:37,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:05:41,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:45,238 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:46,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:51,809 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:05:52,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:05:52,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:05:52,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:05:52,894 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:05:55,296 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:05:56,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:01,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:03,360 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:07,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:07,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:06:07,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:06:07,502 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:06:12,422 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:13,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:18,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:22,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:06:22,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:06:22,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:06:22,487 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:22,894 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:06:23,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:28,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:32,560 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:34,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:37,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:06:37,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:06:37,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:06:39,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:42,643 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:45,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:50,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:06:52,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:06:52,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:06:52,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:06:52,713 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:06:52,895 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:06:55,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:00,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:02,788 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:06,872 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:07,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:07:07,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:07:07,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:07:11,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:07:11,868 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:07:11,869 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:07:11,871 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:07:11,872 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:12,860 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:12,863 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:07:13,865 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:16,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:22,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:22,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:07:22,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:07:22,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:07:22,896 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:07:22,927 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:27,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:32,998 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:33,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:37,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:07:37,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:07:37,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:07:38,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:43,063 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:44,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:49,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:52,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:07:52,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:07:52,483 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:07:52,896 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:07:53,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:07:54,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:07:59,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:03,232 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:05,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:07,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:08:07,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:08:07,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:08:10,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:13,307 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:16,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:21,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:22,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:08:22,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:08:22,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:08:22,896 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:08:23,371 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:27,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:31,431 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:33,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:37,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:08:37,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:08:37,488 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:08:38,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:41,515 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:44,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:49,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:51,583 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:08:52,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:08:52,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:08:52,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:08:52,897 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:08:54,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:08:59,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:00,649 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:05,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:08,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:09:08,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:09:08,780 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:09:10,712 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:11,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:16,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:20,785 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:21,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:22,897 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:09:23,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:09:23,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:09:23,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:09:26,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:30,868 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:32,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:37,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:38,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:09:38,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:09:38,778 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:09:40,941 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:43,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:48,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:51,011 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:09:52,899 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:09:53,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:09:53,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:09:53,777 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:09:54,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:09:59,038 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:01,080 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:04,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:08,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:10:08,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:10:08,779 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:10:09,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:11,151 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:14,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:19,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:21,233 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:22,899 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:10:23,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:10:23,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:10:23,778 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:10:25,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:29,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:10:29,787 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:10:29,787 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:10:29,789 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:10:30,294 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:10:30,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:31,299 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:32,302 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:35,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:38,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:10:38,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:10:38,793 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:10:41,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:41,365 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:46,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:51,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:10:52,899 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:10:53,447 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:10:53,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:10:53,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:10:53,791 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:10:57,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:01,503 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:02,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:07,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:09,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:11:09,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:11:09,545 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:11:13,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:13,591 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:18,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:22,901 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:11:23,692 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:23,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:24,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:11:24,546 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:11:24,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:11:29,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:33,775 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:35,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:39,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:11:39,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:11:39,549 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:11:40,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:42,851 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:46,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:51,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:11:52,911 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:11:54,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:11:54,547 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:11:54,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:11:54,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:11:56,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:01,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:05,007 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:12:06,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:09,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:12:09,547 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:12:09,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:12:12,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:15,072 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:12:18,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:22,902 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:12:23,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:24,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:12:24,547 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:12:24,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:12:25,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:12:29,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:35,215 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:12:35,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:39,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:12:39,547 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:12:39,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:12:40,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:46,014 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:47,288 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:12:51,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:52,904 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:12:54,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:12:54,548 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:12:54,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:12:56,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:12:57,363 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:01,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:06,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:07,434 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:09,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:13:09,548 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:13:09,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:13:12,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:17,507 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:17,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:22,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:22,904 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:13:24,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:13:24,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:13:24,756 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:13:27,572 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:27,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:32,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:37,682 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:38,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:39,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:13:39,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:13:39,754 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:13:43,953 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:47,753 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:13:49,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:52,905 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:13:54,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:13:55,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:13:55,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:13:55,003 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:13:56,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:13:56,507 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:13:56,508 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:13:56,510 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:13:56,975 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:13:57,980 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:00,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:05,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:07,046 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:09,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:14:09,999 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:14:10,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:14:11,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:16,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:17,113 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:21,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:22,905 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:14:25,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:14:25,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:14:25,825 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:14:27,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:29,208 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:32,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:37,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:39,289 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:40,821 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:14:40,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:14:40,822 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:14:43,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:48,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:49,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:52,906 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:14:53,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:14:56,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:14:56,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:14:56,672 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:14:59,442 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:14:59,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:04,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:09,516 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:15:10,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:11,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:15:11,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:15:11,665 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:15:15,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:19,578 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:15:21,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:22,907 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:15:26,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:27,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:15:27,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:15:27,566 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:15:29,649 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:15:32,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:37,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:41,725 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:15:42,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:15:42,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:15:42,563 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:15:43,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:48,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:51,795 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:15:52,907 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:15:53,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:15:58,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:15:58,375 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:15:58,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:15:59,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:01,866 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:04,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:09,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:11,940 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:13,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:16:13,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:16:13,376 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:16:15,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:20,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:22,004 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:22,908 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:16:25,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:28,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:16:28,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:16:28,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:16:31,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:32,064 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:36,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:42,134 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:42,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:43,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:16:43,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:16:43,404 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:16:48,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:52,202 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:16:52,909 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:16:53,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:16:58,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:16:58,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:16:58,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:16:59,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:01,279 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:05,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:10,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:11,345 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:13,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:17:13,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:17:13,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:17:16,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:19,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:17:19,846 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:17:19,847 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:17:19,850 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:17:20,402 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:17:21,407 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:21,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:22,412 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:22,909 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:17:26,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:28,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:17:28,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:17:28,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:17:31,470 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:32,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:37,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:41,537 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:43,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:43,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:17:43,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:17:43,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:17:48,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:51,603 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:17:52,909 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:17:53,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:17:58,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:17:58,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:17:58,556 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:17:59,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:01,673 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:04,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:10,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:11,742 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:13,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:18:13,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:18:13,556 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:18:15,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:21,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:21,811 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:22,910 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:18:26,911 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:28,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:18:28,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:18:28,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:18:32,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:35,898 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:37,984 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:42,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:43,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:18:43,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:18:43,928 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:18:45,972 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:48,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:52,911 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:18:53,912 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:56,050 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:18:58,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:18:58,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:18:58,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:18:58,926 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:19:04,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:08,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:10,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:13,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:19:13,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:19:13,929 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:19:15,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:16,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:20,798 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:22,912 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:19:26,314 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:26,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:28,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:19:28,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:19:28,926 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:19:32,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:36,402 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:38,110 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:43,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:43,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:19:43,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:19:43,927 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:19:45,461 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:48,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:52,928 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:19:53,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:55,525 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:19:58,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:19:58,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:19:58,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:19:58,926 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:20:03,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:05,586 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:08,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:13,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:13,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:20:13,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:20:13,926 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:20:15,649 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:19,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:22,914 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:20:24,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:25,715 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:28,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:20:28,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:20:28,927 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:20:30,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:35,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:35,790 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:40,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:43,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:20:43,814 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:20:43,815 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:20:43,818 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:20:43,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:20:43,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:20:43,938 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:20:43,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:20:44,859 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:45,865 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:46,113 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:51,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:52,915 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:20:55,935 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:20:56,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:20:58,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:20:58,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:20:58,937 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:21:02,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:05,992 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:07,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:12,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:13,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:21:13,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:21:13,939 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:21:16,076 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:18,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:22,915 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:21:23,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:24,135 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:28,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:28,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:21:28,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:21:28,937 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:21:33,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:34,219 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:38,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:43,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:21:43,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:21:43,937 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:21:44,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:44,289 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:49,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:52,915 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:21:54,349 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:21:54,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:21:58,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:21:58,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:21:58,938 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:22:00,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:04,414 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:05,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:10,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:13,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:22:13,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:22:13,938 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:22:14,500 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:16,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:21,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:22,916 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:22:24,564 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:26,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:28,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:22:28,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:22:28,938 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:22:32,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:34,625 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:37,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:42,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:43,696 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:43,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:22:43,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:22:43,939 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:22:48,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:52,917 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:22:53,759 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:22:53,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:58,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:22:58,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:22:58,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:22:58,942 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:23:03,827 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:04,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:09,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:13,893 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:13,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:23:13,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:23:13,939 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:23:15,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:20,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:21,963 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:22,917 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:23:25,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:29,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:23:29,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:23:29,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:23:31,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:32,029 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:36,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:41,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:42,093 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:44,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:23:44,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:23:44,373 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:23:47,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:52,159 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:23:52,917 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:23:52,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:57,920 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:23:59,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:23:59,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:23:59,376 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:24:00,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:24:00,066 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:24:00,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:24:00,068 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:24:00,212 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:24:01,217 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:02,223 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:03,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:08,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:12,296 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:13,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:14,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:24:14,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:24:14,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:24:19,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:22,358 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:22,918 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:24:24,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:29,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:24:29,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:24:29,373 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:24:30,413 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:30,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:35,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:40,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:41,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:44,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:24:44,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:24:44,373 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:24:46,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:50,541 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:24:52,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:52,919 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:24:57,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:24:59,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:24:59,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:24:59,373 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:25:00,605 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:03,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:08,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:10,681 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:13,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:14,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:25:14,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:25:14,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:25:19,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:20,747 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:22,920 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:25:24,921 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:29,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:25:29,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:25:29,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:25:30,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:30,817 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:35,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:39,882 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:41,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:44,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:25:44,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:25:44,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:25:46,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:49,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:25:52,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:52,920 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:25:57,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:25:59,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:25:59,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:25:59,374 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:26:00,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:03,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:09,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:10,078 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:14,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:15,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:26:15,932 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:26:15,933 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:26:18,135 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:20,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:22,921 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:26:25,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:28,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:30,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:26:30,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:26:30,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:30,932 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:26:36,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:38,272 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:41,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:46,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:26:46,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:26:46,317 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:26:46,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:48,344 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:26:51,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:52,921 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:26:56,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:26:58,410 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:01,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:27:01,314 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:27:01,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:27:02,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:07,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:08,475 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:12,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:16,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:27:16,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:27:16,881 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:27:18,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:18,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:27:18,268 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:27:18,269 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:27:18,270 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:27:18,577 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:27:20,586 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:21,591 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:22,922 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:27:23,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:29,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:30,652 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:31,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:27:31,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:27:31,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:27:35,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:40,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:40,721 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:45,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:46,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:27:46,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:27:46,880 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:27:50,779 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:27:51,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:27:52,922 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:27:56,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:00,853 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:01,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:28:01,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:28:01,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:28:02,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:07,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:10,918 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:12,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:17,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:28:17,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:28:17,386 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:28:18,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:20,993 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:22,923 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:28:23,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:29,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:31,054 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:32,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:28:32,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:28:32,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:28:34,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:40,116 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:40,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:45,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:47,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:28:47,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:28:47,715 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:28:50,196 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:28:51,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:28:52,923 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:28:56,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:00,260 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:02,711 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:29:02,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:29:02,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:02,713 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:29:08,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:10,327 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:13,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:17,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:29:17,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:29:17,715 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:29:18,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:20,404 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:22,924 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:29:24,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:30,469 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:30,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:32,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:29:32,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:29:32,713 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:29:35,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:40,551 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:41,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:46,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:47,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:29:47,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:29:47,714 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:29:50,613 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:29:52,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:29:52,925 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:29:57,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:00,681 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:02,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:30:02,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:30:02,713 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:30:03,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:08,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:08,735 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:13,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:17,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:30:17,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:30:17,715 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:30:19,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:20,814 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:22,925 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:30:24,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:29,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:30,883 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:32,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:30:32,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:30:32,713 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:30:34,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:37,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:30:37,717 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:30:37,717 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:30:37,723 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:30:37,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:30:38,944 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:40,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:45,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:47,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:30:47,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:30:47,714 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:30:50,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:30:51,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:30:52,926 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:30:56,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:01,083 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:01,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:02,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:31:02,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:31:02,714 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:31:07,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:11,165 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:13,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:17,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:31:17,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:31:17,716 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:31:18,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:21,253 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:22,926 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:31:24,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:30,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:30,325 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:32,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:31:32,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:31:32,714 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:31:35,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:40,383 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:41,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:46,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:48,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:31:48,272 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:31:48,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:31:50,452 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:31:52,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:31:52,927 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:31:57,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:02,527 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:03,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:03,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:32:03,272 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:32:03,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:32:08,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:12,600 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:14,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:18,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:32:18,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:32:18,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:32:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:22,665 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:22,927 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:32:25,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:30,727 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:31,842 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:33,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:32:33,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:32:33,990 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:32:37,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:40,806 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:42,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:47,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:48,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:32:48,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:32:48,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:32:52,875 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:32:52,928 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:32:53,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:32:58,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:00,940 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:03,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:03,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:33:03,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:33:03,992 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:33:09,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:14,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:17,049 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:18,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:33:18,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:33:18,992 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:33:20,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:22,930 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:33:25,835 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:27,118 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:30,836 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:33,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:33:33,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:33:33,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:33:36,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:37,183 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:41,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:46,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:47,251 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:48,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:33:48,992 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:33:48,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:33:52,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:33:52,933 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:33:57,313 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:33:57,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:02,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:03,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:34:03,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:34:03,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:34:04,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:34:04,647 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:34:04,648 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:34:04,650 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:34:05,363 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:34:07,372 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:08,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:13,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:17,448 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:18,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:34:18,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:34:18,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:34:19,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:22,933 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:34:25,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:27,512 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:30,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:33,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:34:33,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:34:33,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:34:34,588 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:35,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:40,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:44,648 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:46,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:48,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:34:48,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:34:48,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:34:51,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:34:52,933 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:34:54,710 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:34:56,915 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:01,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:03,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:35:03,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:35:03,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:35:04,775 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:07,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:12,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:14,839 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:17,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:18,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:35:18,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:35:18,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:35:22,934 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:35:22,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:24,916 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:28,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:33,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:33,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:35:33,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:35:33,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:35:34,987 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:39,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:45,053 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:45,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:48,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:35:48,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:35:48,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:35:50,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:35:52,934 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:35:55,122 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:35:55,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:00,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:03,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:36:03,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:36:03,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:36:05,191 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:36:06,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:11,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:13,258 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:36:16,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:18,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:36:18,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:36:18,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:36:22,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:22,935 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:36:27,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:32,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:33,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:36:33,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:36:34,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:36:37,392 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:36:38,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:43,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:47,449 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:36:48,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:49,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:36:49,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:36:49,046 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:36:52,936 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:36:54,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:36:57,540 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:36:59,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:04,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:37:04,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:37:04,009 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:37:05,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:07,623 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:37:10,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:15,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:17,723 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:37:19,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:37:19,006 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:37:19,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:37:21,244 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:22,937 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:37:26,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:31,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:34,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:37:34,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:37:34,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:37:37,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:38,840 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:37:42,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:47,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:48,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:37:48,339 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:37:48,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:37:48,342 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:37:48,908 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:37:48,908 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:37:49,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:37:49,006 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:37:49,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:37:49,914 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:37:52,937 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:37:52,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:57,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:37:58,979 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:38:03,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:04,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:38:04,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:38:04,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:38:08,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:09,050 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:38:13,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:18,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:19,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:38:19,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:38:19,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:38:21,120 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:38:22,938 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:38:23,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:28,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:31,180 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:38:33,996 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:34,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:38:34,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:38:34,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:38:39,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:41,278 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:38:44,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:49,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:38:49,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:38:49,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:38:50,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:38:52,946 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:38:55,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:00,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:04,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:39:04,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:39:04,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:39:06,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:11,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:11,551 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:39:16,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:19,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:39:19,008 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:39:19,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:39:21,609 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:39:22,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:22,946 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:39:27,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:31,681 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:39:33,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:34,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:39:34,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:39:34,009 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:39:39,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:43,774 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:39:44,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:49,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:50,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:39:50,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:39:50,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:39:52,947 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:39:53,844 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:39:54,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:39:59,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:03,922 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:05,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:40:05,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:05,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:40:05,525 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:40:10,690 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:12,994 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:16,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:20,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:40:20,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:40:20,850 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:40:22,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:22,947 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:40:23,077 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:27,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:33,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:33,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:35,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:40:35,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:40:35,849 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:40:39,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:44,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:45,211 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:49,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:51,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:40:51,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:40:51,502 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:40:52,948 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:40:54,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:40:55,283 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:40:59,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:05,375 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:05,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:06,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:41:06,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:41:06,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:41:10,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:15,460 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:16,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:21,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:21,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:41:21,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:41:21,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:41:22,949 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:41:25,530 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:27,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:32,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:33,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:41:33,496 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:41:33,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:41:33,500 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:41:33,609 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:41:35,620 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:36,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:41:36,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:41:36,812 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:41:38,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:43,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:45,683 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:48,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:51,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:41:51,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:41:51,813 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:41:52,950 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:41:54,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:41:55,767 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:41:59,038 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:04,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:05,934 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:06,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:42:06,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:42:06,812 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:42:10,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:15,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:16,002 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:20,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:22,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:42:22,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:42:22,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:42:22,951 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:42:25,267 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:25,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:30,858 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:35,614 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:36,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:37,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:42:37,453 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:42:37,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:42:41,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:45,683 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:46,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:51,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:42:52,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:42:52,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:42:52,906 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:42:53,069 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:42:55,763 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:42:57,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:02,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:05,833 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:07,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:07,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:43:07,901 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:43:07,901 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:43:13,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:15,898 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:18,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:22,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:43:22,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:43:22,939 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:43:23,094 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:43:24,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:25,987 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:29,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:34,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:36,063 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:37,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:43:37,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:43:37,935 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:43:40,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:45,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:46,172 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:50,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:43:52,956 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:43:53,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:43:53,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:43:53,311 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:43:56,254 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:43:56,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:01,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:07,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:07,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:08,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:44:08,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:44:08,307 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:44:12,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:17,415 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:18,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:22,957 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:44:23,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:44:23,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:44:23,869 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:44:24,086 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:27,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:29,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:34,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:37,548 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:38,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:44:38,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:44:38,865 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:44:40,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:45,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:47,615 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:50,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:52,959 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:44:54,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:44:54,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:44:54,138 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:44:55,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:44:55,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:44:55,606 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:44:55,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:44:55,608 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:44:55,675 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:44:57,690 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:44:58,695 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:00,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:05,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:07,785 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:09,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:45:09,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:45:09,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:45:11,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:16,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:17,859 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:21,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:22,961 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:45:24,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:45:24,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:45:24,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:45:27,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:27,927 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:32,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:37,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:38,012 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:39,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:45:39,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:45:39,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:45:42,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:47,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:48,086 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:45:52,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:52,960 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:45:55,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:45:55,680 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:45:55,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:45:58,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:45:58,152 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:03,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:08,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:10,241 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:10,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:46:10,676 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:46:10,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:46:13,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:19,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:20,409 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:22,961 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:46:24,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:26,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:46:26,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:46:26,009 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:46:27,467 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:30,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:35,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:39,641 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:40,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:41,006 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:46:41,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:46:41,007 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:46:46,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:49,705 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:46:51,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:52,962 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:46:56,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:46:56,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:46:56,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:46:56,847 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:46:59,782 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:02,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:07,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:09,853 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:11,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:47:11,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:47:11,844 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:47:13,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:18,004 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:19,925 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:22,962 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:47:23,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:27,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:47:27,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:47:27,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:47:29,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:30,067 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:34,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:40,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:40,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:42,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:47:42,474 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:47:42,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:47:46,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:50,254 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:47:51,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:52,963 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:47:56,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:47:57,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:47:57,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:47:57,474 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:48:00,310 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:02,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:07,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:10,387 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:12,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:48:12,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:48:12,473 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:48:12,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:17,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:18,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:48:18,457 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:48:18,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:48:18,459 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:48:19,445 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:48:20,451 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:22,964 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:48:22,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:27,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:28,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:48:28,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:48:28,164 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:48:32,538 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:33,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:38,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:41,628 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:43,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:48:43,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:48:43,189 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:48:44,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:49,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:52,964 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:48:53,878 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:48:54,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:48:58,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:48:58,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:48:58,163 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:49:00,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:05,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:09,987 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:49:10,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:13,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:49:13,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:49:13,163 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:49:16,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:20,066 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:49:21,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:22,965 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:49:26,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:28,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:49:28,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:49:28,165 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:49:30,135 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:49:32,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:37,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:40,211 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:49:43,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:49:43,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:43,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:49:43,164 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:49:48,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:50,294 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:49:52,966 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:49:54,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:49:58,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:49:58,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:49:58,164 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:50:00,357 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:00,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:05,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:10,423 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:10,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:13,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:50:13,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:50:13,165 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:50:16,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:20,503 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:21,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:22,966 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:50:26,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:28,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:50:28,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:50:28,165 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:50:30,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:32,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:37,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:40,644 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:42,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:43,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:50:43,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:50:43,164 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:50:48,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:50,712 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:50:52,967 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:50:53,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:50:58,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:50:58,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:50:58,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:50:59,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:00,790 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:04,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:09,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:10,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:13,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:51:13,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:51:13,164 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:51:15,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:19,941 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:20,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:22,967 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:51:25,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:28,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:51:28,367 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:51:28,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:51:31,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:32,020 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:36,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:41,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:42,114 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:43,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:51:43,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:51:43,366 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:51:47,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:49,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:51:49,627 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:51:49,627 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:51:49,629 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:51:50,169 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:51:51,174 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:52,177 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:51:52,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:52,968 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:51:57,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:51:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:51:58,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:51:58,369 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:52:00,241 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:03,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:08,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:12,351 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:13,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:52:13,366 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:52:13,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:52:13,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:18,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:22,427 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:22,968 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:52:23,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:28,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:52:28,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:52:28,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:52:29,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:32,502 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:34,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:39,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:42,598 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:43,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:52:43,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:52:43,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:52:44,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:50,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:52,681 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:52:52,969 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:52:55,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:52:58,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:52:58,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:52:58,526 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:53:01,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:01,954 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:06,796 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:12,032 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:12,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:13,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:53:13,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:53:13,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:53:17,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:22,151 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:22,969 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:53:22,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:27,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:29,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:53:29,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:53:29,326 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:53:32,227 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:33,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:38,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:42,297 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:43,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:44,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:53:44,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:53:44,320 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:53:49,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:52,369 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:53:52,977 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:53:54,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:53:59,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:53:59,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:53:59,322 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:54:00,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:02,435 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:05,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:10,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:12,497 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:14,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:54:14,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:54:14,321 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:54:16,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:22,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:22,585 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:22,975 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:54:27,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:29,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:54:29,833 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:54:29,834 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:54:32,654 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:33,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:38,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:42,719 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:44,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:44,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:54:44,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:54:44,794 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:54:50,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:54:52,977 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:54:54,808 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:54:55,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:00,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:55:00,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:55:00,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:55:01,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:04,879 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:06,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:11,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:11,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:55:11,775 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:55:11,776 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:55:11,778 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:55:11,935 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:55:12,940 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:14,953 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:15,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:55:15,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:55:15,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:55:17,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:22,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:22,978 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:55:25,027 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:28,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:30,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:55:30,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:55:30,563 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:55:33,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:37,114 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:39,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:44,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:45,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:55:45,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:55:45,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:55:46,170 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:49,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:52,980 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:55:54,842 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:55:56,240 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:55:59,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:00,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:56:00,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:56:00,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:56:04,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:06,310 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:56:09,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:14,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:15,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:56:15,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:56:15,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:56:16,382 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:56:20,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:22,982 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:56:25,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:28,466 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:56:30,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:56:30,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:56:30,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:56:31,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:37,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:38,534 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:56:42,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:45,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:56:45,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:56:45,564 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:56:47,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:48,604 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:56:52,213 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:52,981 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:56:57,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:56:58,678 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:00,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:57:00,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:57:00,563 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:57:02,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:07,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:08,755 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:12,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:15,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:57:15,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:57:15,564 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:57:18,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:18,822 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:22,982 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:57:23,983 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:28,900 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:29,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:30,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:57:30,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:57:30,563 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:57:34,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:38,963 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:40,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:45,244 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:45,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:57:45,915 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:57:45,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:57:49,041 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:57:51,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:52,983 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:57:56,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:57:59,105 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:00,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:58:00,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:58:00,914 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:58:02,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:07,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:09,173 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:12,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:15,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:58:15,915 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:58:15,916 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:58:18,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:19,243 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:22,984 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:58:23,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:29,313 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:29,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:30,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:58:30,914 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:58:30,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:58:35,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:37,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 16:58:37,308 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 16:58:37,312 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 16:58:37,314 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 16:58:37,365 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 16:58:39,376 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:40,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:45,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:46,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:58:46,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:58:46,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:58:48,449 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:58:50,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:52,995 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:58:55,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:58:58,523 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:01,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:59:01,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:59:01,066 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:59:01,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:06,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:08,588 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:12,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:16,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:59:16,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:59:16,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:59:18,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:18,658 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:22,985 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:59:23,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:28,718 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:29,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:31,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:59:31,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:59:31,066 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:59:35,240 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:38,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:40,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:45,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:46,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 16:59:46,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 16:59:46,355 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 16:59:48,867 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 16:59:50,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:52,985 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 16:59:55,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 16:59:58,943 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:01,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:00:01,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:00:01,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:01,352 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:00:06,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:09,009 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:12,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:16,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:00:16,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:00:16,438 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:00:17,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:19,117 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:22,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:22,986 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:00:28,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:29,187 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:31,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:00:31,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:00:31,437 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:00:34,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:39,262 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:39,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:44,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:46,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:00:46,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:00:46,488 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:00:49,335 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:00:49,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:52,987 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:00:54,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:00:59,425 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:00,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:01,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:01:01,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:01:01,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:01:05,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:10,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:11,506 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:15,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:17,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:01:17,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:01:17,791 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:01:21,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:21,581 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:22,987 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:01:26,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:30,644 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:32,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:01:32,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:01:32,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:01:33,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:38,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:42,740 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:43,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:48,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:49,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:01:49,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:01:49,150 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:01:52,825 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:01:52,988 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:01:53,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:01:58,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:01,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:02:01,079 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:02:01,080 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:02:01,082 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:02:01,902 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:02:02,906 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:03,910 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:04,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:04,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:02:04,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:02:04,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:02:09,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:12,982 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:14,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:19,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:20,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:02:20,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:02:20,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:02:22,988 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:02:23,048 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:24,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:29,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:33,113 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:35,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:35,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:02:35,760 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:02:35,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:02:40,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:45,192 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:46,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:51,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:02:51,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:02:51,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:02:51,730 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:02:52,989 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:02:55,263 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:02:56,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:01,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:05,328 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:03:06,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:03:06,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:03:06,726 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:03:07,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:12,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:15,393 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:03:17,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:21,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:03:21,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:03:21,730 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:03:22,991 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:03:22,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:25,486 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:03:27,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:33,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:35,549 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:03:36,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:03:36,726 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:03:36,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:03:38,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:44,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:45,618 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:03:49,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:03:51,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:03:51,726 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:03:51,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:03:52,992 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:03:54,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:00,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:01,724 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:04:05,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:08,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:04:08,004 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:04:08,006 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:04:11,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:11,788 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:04:16,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:21,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:22,992 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:04:23,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:04:23,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:04:23,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:04:26,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:31,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:36,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:38,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:04:38,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:04:38,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:04:41,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:46,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:49,004 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:04:51,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:52,993 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:04:53,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:04:53,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:04:53,011 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:04:56,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:04:59,073 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:01,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:07,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:08,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:05:08,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:05:08,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:05:09,146 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:13,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:18,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:19,231 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:22,993 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:05:23,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:05:23,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:05:23,011 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:05:24,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:29,299 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:29,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:34,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:38,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:05:38,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:05:38,011 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:05:39,403 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:40,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:45,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:49,460 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:05:51,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:52,994 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:05:53,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:05:53,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:05:53,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:05:56,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:05:58,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:05:58,467 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:05:58,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:05:58,470 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:05:58,525 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:05:59,532 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:01,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:06,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:08,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:06:08,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:06:08,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:06:09,634 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:12,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:17,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:21,721 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:22,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:22,994 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:06:23,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:06:23,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:06:23,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:06:28,209 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:31,792 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:33,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:38,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:38,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:06:38,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:06:38,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:06:41,862 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:43,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:48,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:06:51,932 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:06:52,997 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:06:53,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:06:53,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:06:53,429 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:06:54,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:00,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:01,009 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:05,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:08,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:07:08,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:07:08,432 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:07:11,069 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:11,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:16,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:21,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:22,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:22,995 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:07:23,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:07:23,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:07:23,430 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:07:27,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:31,213 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:33,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:38,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:39,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:07:39,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:07:39,427 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:07:41,282 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:43,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:48,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:51,346 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:07:52,996 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:07:53,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:07:54,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:07:54,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:07:54,425 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:07:59,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:01,416 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:04,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:09,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:08:09,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:08:09,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:08:09,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:11,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:15,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:20,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:21,542 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:22,996 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:08:24,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:08:24,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:08:24,511 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:08:25,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:30,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:33,626 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:36,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:40,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:08:40,276 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:08:40,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:08:41,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:43,725 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:46,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:52,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:08:52,997 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:08:53,800 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:08:55,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:08:55,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:08:55,275 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:08:57,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:02,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:03,862 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:07,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:10,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:09:10,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:09:10,658 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:09:12,868 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:13,929 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:17,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:22,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:09:22,514 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:09:22,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:09:22,517 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:09:22,998 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:09:23,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:23,019 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:09:24,024 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:25,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:09:25,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:09:25,656 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:09:28,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:34,090 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:34,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:39,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:41,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:09:41,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:09:41,312 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:09:44,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:45,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:50,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:52,999 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:09:53,218 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:09:56,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:09:56,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:09:56,308 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:09:56,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:10:01,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:03,277 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:06,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:11,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:12,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:10:12,213 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:10:12,214 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:10:15,363 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:17,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:22,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:22,999 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:10:25,446 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:27,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:10:27,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:10:27,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:10:28,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:34,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:35,515 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:39,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:42,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:10:42,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:10:42,216 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:10:43,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:45,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:50,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:53,000 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:10:53,652 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:10:56,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:10:57,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:10:57,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:10:57,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:11:01,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:05,781 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:06,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:11,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:12,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:11:12,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:11:12,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:11:15,842 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:16,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:21,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:23,001 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:11:25,908 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:27,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:11:27,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:11:27,392 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:11:27,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:32,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:35,984 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:37,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:42,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:11:42,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:11:42,716 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:11:42,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:46,059 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:48,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:53,003 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:11:54,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:11:56,118 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:11:57,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:11:57,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:11:57,715 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:11:59,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:05,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:06,195 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:10,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:13,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:12:13,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:12:13,737 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:12:16,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:16,261 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:21,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:23,004 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:12:26,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:28,348 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:28,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:12:28,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:12:28,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:12:32,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:37,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:37,412 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:42,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:44,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:12:44,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:12:44,358 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:12:45,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:12:45,787 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:12:45,787 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:12:45,788 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:12:46,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:12:47,486 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:47,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:48,491 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:52,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:53,005 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:12:57,558 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:12:57,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:12:59,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:12:59,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:12:59,354 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:13:03,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:07,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:13:08,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:13,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:14,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:13:14,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:13:14,510 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:13:17,705 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:13:18,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:23,007 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:13:24,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:27,773 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:13:29,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:13:29,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:13:29,509 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:13:29,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:34,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:37,859 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:13:40,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:45,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:13:45,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:13:45,285 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:13:45,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:47,955 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:13:50,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:53,007 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:13:56,008 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:13:58,021 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:00,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:14:00,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:14:00,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:14:01,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:06,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:08,097 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:12,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:16,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:14:16,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:14:16,029 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:14:17,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:20,182 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:22,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:23,008 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:14:27,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:30,256 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:31,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:14:31,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:14:31,026 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:14:33,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:39,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:40,336 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:44,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:46,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:14:46,632 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:14:46,632 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:14:50,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:14:50,411 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:14:53,067 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:14:55,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:00,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:00,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:01,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:15:01,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:15:01,630 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:15:05,835 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:10,553 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:11,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:16,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:18,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:15:18,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:15:18,142 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:15:21,636 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:21,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:23,008 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:15:27,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:31,704 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:32,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:33,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:15:33,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:15:33,142 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:15:37,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:41,779 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:42,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:47,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:48,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:15:48,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:15:48,723 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:15:51,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:15:52,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:15:53,009 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:15:58,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:01,926 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:03,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:03,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:16:03,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:16:03,720 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:16:08,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:10,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:16:10,175 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:16:10,176 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:16:10,178 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:16:10,995 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:16:12,002 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:14,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:18,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:16:18,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:16:18,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:16:19,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:22,070 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:23,009 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:16:25,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:30,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:32,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:33,757 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:16:33,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:16:33,759 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:16:35,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:41,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:42,225 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:46,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:48,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:16:48,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:16:48,761 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:16:51,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:16:52,292 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:16:53,010 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:16:57,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:02,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:02,363 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:03,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:17:03,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:17:03,759 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:17:08,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:12,430 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:13,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:18,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:19,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:17:19,123 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:17:19,124 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:17:22,516 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:23,011 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:17:24,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:29,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:32,603 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:34,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:17:34,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:17:34,123 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:17:34,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:39,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:42,683 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:44,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:49,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:17:49,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:17:49,439 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:17:49,618 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:17:52,755 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:17:53,011 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:17:55,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:00,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:02,828 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:04,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:18:04,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:18:04,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:18:05,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:10,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:11,941 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:16,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:20,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:18:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:18:20,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:18:21,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:22,014 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:23,012 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:18:27,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:32,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:32,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:35,113 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:18:35,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:18:35,114 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:18:37,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:42,179 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:42,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:47,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:50,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:18:50,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:18:50,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:18:52,259 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:18:53,013 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:18:53,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:18:58,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:02,345 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:19:03,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:05,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:19:05,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:19:05,114 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:19:09,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:12,504 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:19:14,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:19,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:20,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:19:20,123 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:19:20,124 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:19:23,014 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:19:25,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:28,651 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:19:30,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:35,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:19:35,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:19:35,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:19:36,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:36,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:19:36,415 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:19:36,415 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:19:36,422 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:19:36,705 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:19:38,716 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:19:41,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:47,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:48,835 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:19:50,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:19:50,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:19:50,115 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:19:52,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:53,016 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:19:58,082 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:19:58,905 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:03,083 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:05,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:20:05,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:20:05,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:20:08,985 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:09,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:14,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:18,058 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:20,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:20,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:20:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:20:20,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:20:23,017 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:20:25,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:28,119 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:31,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:35,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:20:35,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:20:35,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:20:36,179 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:36,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:41,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:46,239 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:47,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:50,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:20:50,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:20:50,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:20:53,019 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:20:53,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:20:56,309 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:20:58,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:03,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:05,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:21:05,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:21:05,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:21:06,384 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:21:09,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:14,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:16,452 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:21:19,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:20,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:21:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:21:20,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:21:23,020 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:21:25,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:26,520 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:21:30,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:35,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:21:35,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:21:35,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:21:35,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:36,587 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:21:40,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:45,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:46,655 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:21:50,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:21:50,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:21:50,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:21:51,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:53,022 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:21:56,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:21:56,738 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:01,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:05,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:22:05,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:22:05,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:22:06,813 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:07,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:12,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:16,878 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:18,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:20,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:22:20,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:22:20,117 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:22:23,023 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:22:24,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:26,938 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:29,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:34,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:35,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:22:35,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:22:35,119 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:22:37,008 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:40,679 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:46,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:47,076 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:22:50,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:22:50,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:22:50,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:22:52,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:53,024 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:22:55,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:22:55,983 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:22:55,983 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:22:55,986 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:22:56,134 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:22:57,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:22:59,149 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:02,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:05,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:23:05,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:23:05,121 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:23:07,209 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:08,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:13,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:16,281 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:19,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:20,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:23:20,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:23:20,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:23:23,030 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:23:25,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:26,354 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:30,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:35,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:23:35,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:23:35,120 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:23:36,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:38,439 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:41,200 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:46,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:48,554 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:23:50,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:23:50,118 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:23:50,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:23:51,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:53,027 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:23:56,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:23:58,624 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:01,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:05,264 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:24:05,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:24:05,266 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:24:06,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:08,689 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:11,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:17,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:18,757 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:20,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:24:20,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:24:20,264 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:24:22,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:23,029 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:24:28,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:28,829 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:33,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:35,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:24:35,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:24:35,976 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:24:38,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:38,893 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:43,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:48,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:48,963 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:50,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:24:50,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:24:50,972 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:24:53,030 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:24:54,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:24:59,056 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:24:59,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:04,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:05,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:25:05,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:25:05,973 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:25:09,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:25:09,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:14,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:19,227 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:25:20,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:20,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:25:20,972 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:25:21,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:25:23,035 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:25:26,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:29,292 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:25:31,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:35,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:25:35,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:25:35,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:25:36,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:39,353 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:25:41,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:46,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:48,408 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:25:50,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:25:50,973 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:25:50,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:25:52,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:53,030 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:25:57,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:25:58,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:02,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:05,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:26:05,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:26:05,976 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:26:08,555 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:08,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:13,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:16,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:26:16,549 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:26:16,549 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:26:16,552 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:26:16,630 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:26:18,642 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:19,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:20,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:26:20,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:26:21,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:26:23,031 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:26:25,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:28,724 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:30,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:35,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:35,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:26:35,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:26:35,976 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:26:38,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:40,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:45,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:48,876 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:26:50,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:26:50,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:50,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:26:50,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:26:53,031 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:26:56,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:26:58,946 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:01,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:05,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:27:05,975 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:27:05,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:27:07,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:09,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:12,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:18,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:19,142 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:20,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:27:20,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:27:20,976 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:27:23,032 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:27:24,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:29,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:29,208 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:34,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:35,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:27:35,976 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:27:35,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:27:39,213 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:39,321 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:44,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:49,479 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:27:50,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:50,974 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:27:50,975 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:27:51,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:27:53,033 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:27:56,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:27:59,540 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:01,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:06,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:06,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:28:06,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:28:06,805 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:28:08,593 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:12,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:17,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:18,678 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:21,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:28:21,801 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:28:21,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:28:23,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:23,033 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:28:28,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:28,747 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:33,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:36,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:28:36,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:28:36,804 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:28:38,835 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:38,921 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:43,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:48,912 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:28:49,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:51,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:28:51,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:28:51,802 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:28:53,034 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:28:55,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:28:58,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:00,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:05,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:07,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:29:07,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:29:07,219 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:29:10,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:11,063 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:15,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:19,130 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:21,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:22,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:29:22,223 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:29:22,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:29:23,034 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:29:27,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:29,202 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:32,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:37,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:29:37,218 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:29:37,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:29:37,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:38,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:29:38,156 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:29:38,157 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:29:38,159 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:29:38,264 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:29:39,271 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:43,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:48,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:49,345 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:52,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:29:52,217 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:29:52,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:29:53,045 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:29:54,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:29:59,411 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:29:59,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:04,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:07,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:30:07,219 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:30:07,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:30:09,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:30:09,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:14,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:19,537 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:30:19,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:22,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:30:22,218 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:30:22,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:30:23,035 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:30:25,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:29,618 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:30:30,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:35,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:37,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:30:37,220 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:30:37,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:30:39,698 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:30:40,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:45,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:51,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:30:51,783 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:30:52,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:30:52,218 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:30:52,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:30:53,036 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:30:57,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:00,852 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:02,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:07,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:08,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:31:08,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:31:08,796 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:31:10,928 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:13,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:18,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:21,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:23,036 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:31:23,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:31:23,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:31:23,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:23,793 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:31:28,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:31,070 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:34,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:39,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:31:39,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:31:39,369 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:31:40,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:43,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:45,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:50,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:31:53,037 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:31:53,193 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:31:54,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:31:54,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:31:54,366 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:31:56,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:01,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:03,266 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:32:06,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:09,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:32:09,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:32:09,995 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:32:12,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:13,329 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:32:17,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:22,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:23,037 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:32:23,396 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:32:24,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:32:24,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:32:24,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:32:28,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:33,467 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:32:34,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:39,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:40,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:32:40,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:32:40,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:32:43,565 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:32:44,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:49,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:53,038 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:32:55,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:32:55,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:32:55,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:32:55,762 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:33:00,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:05,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:10,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:33:10,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:33:10,766 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:33:11,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:13,925 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:16,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:21,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:21,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:33:21,792 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:33:21,793 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:33:21,795 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:33:21,985 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:33:22,989 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:23,038 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:33:23,994 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:25,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:33:25,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:33:25,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:33:27,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:32,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:33,059 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:37,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:41,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:33:41,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:33:41,256 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:33:42,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:43,127 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:47,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:53,039 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:33:53,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:33:53,193 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:33:56,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:33:56,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:33:56,255 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:33:58,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:03,393 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:34:03,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:08,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:11,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:34:11,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:34:11,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:34:13,461 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:34:13,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:18,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:23,039 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:34:24,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:25,548 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:34:26,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:34:26,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:34:26,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:34:29,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:33,610 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:34:34,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:39,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:41,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:34:41,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:34:41,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:34:45,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:49,711 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:34:50,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:53,040 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:34:56,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:34:56,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:34:56,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:34:56,291 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:34:59,783 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:01,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:06,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:09,902 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:11,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:35:11,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:35:11,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:35:12,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:17,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:22,044 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:22,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:23,040 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:35:26,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:35:26,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:35:26,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:35:28,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:33,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:38,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:41,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:41,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:35:41,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:35:41,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:35:43,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:48,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:49,209 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:53,041 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:35:54,042 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:35:57,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:35:57,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:35:57,419 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:35:59,279 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:35:59,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:04,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:10,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:11,370 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:36:12,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:36:12,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:36:12,416 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:36:15,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:21,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:21,455 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:36:23,042 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:36:27,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:27,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:36:27,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:36:27,941 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:36:31,521 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:36:32,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:37,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:41,580 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:36:42,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:36:42,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:36:42,939 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:36:43,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:48,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:51,660 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:36:53,042 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:36:54,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:58,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:36:58,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:36:58,074 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:36:59,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:36:59,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:36:59,466 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:36:59,467 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:36:59,469 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:36:59,712 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:37:01,724 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:04,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:10,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:11,778 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:13,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:37:13,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:37:13,072 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:37:15,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:19,842 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:20,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:23,043 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:37:26,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:28,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:37:28,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:37:28,450 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:37:31,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:31,924 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:36,775 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:41,989 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:42,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:43,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:37:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:37:43,446 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:37:47,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:52,056 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:37:53,044 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:37:53,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:58,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:37:58,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:37:58,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:37:58,448 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:38:00,111 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:03,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:09,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:10,173 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:13,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:38:13,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:38:13,448 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:38:14,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:20,245 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:20,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:23,055 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:38:26,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:28,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:38:28,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:38:28,447 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:38:30,311 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:31,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:36,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:39,370 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:42,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:38:43,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:38:43,447 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:38:47,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:49,438 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:53,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:53,044 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:38:55,473 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:38:58,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:38:58,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:38:58,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:38:58,447 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:39:01,500 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:03,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:08,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:11,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:13,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:39:13,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:39:13,448 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:39:14,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:19,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:21,721 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:23,045 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:39:25,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:29,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:39:29,152 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:39:29,153 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:39:30,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:31,795 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:35,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:41,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:41,862 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:44,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:39:44,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:39:44,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:39:46,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:51,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:51,924 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:39:53,046 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:39:57,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:39:59,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:39:59,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:39:59,149 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:40:02,007 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:02,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:07,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:09,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:40:09,450 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:40:09,450 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:40:09,452 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:40:10,061 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:10,062 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:40:12,071 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:13,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:14,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:40:14,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:40:14,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:40:19,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:22,146 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:23,046 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:40:25,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:29,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:40:29,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:40:29,150 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:40:30,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:32,210 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:35,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:40,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:42,283 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:44,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:40:44,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:40:44,149 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:40:45,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:50,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:52,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:40:53,047 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:40:56,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:40:59,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:40:59,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:40:59,609 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:41:01,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:02,422 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:06,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:12,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:12,492 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:14,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:41:14,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:41:14,604 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:41:17,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:21,556 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:23,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:23,048 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:41:28,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:29,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:41:29,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:41:29,608 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:41:31,638 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:33,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:38,694 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:41,705 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:44,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:44,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:41:44,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:41:44,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:41:49,809 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:51,794 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:41:53,048 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:41:55,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:41:59,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:41:59,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:41:59,609 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:42:00,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:01,865 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:05,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:10,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:11,935 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:14,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:42:14,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:42:14,605 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:42:15,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:21,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:22,014 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:23,048 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:42:27,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:29,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:42:29,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:42:29,608 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:42:32,078 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:32,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:37,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:42,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:43,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:44,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:42:44,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:42:44,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:42:48,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:52,246 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:42:53,049 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:42:54,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:59,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:42:59,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:42:59,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:42:59,608 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:43:02,323 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:04,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:09,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:12,396 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:14,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:14,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:43:14,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:43:14,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:43:19,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:22,472 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:23,049 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:43:25,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:29,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:43:29,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:43:29,608 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:43:30,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:30,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:43:30,456 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:43:30,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:43:30,458 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:43:30,549 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:43:32,561 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:35,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:41,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:42,630 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:44,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:43:44,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:43:44,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:43:46,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:52,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:43:52,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:43:53,050 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:43:58,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:00,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:44:00,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:44:00,162 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:44:01,770 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:03,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:08,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:13,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:13,874 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:15,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:44:15,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:44:15,160 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:44:19,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:23,051 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:44:23,974 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:25,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:30,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:30,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:44:30,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:44:30,500 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:44:34,038 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:35,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:40,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:44,108 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:45,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:44:45,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:44:45,497 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:44:46,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:51,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:44:53,051 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:44:54,188 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:44:57,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:00,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:45:00,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:45:00,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:45:03,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:04,268 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:08,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:13,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:14,347 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:15,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:45:15,803 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:45:15,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:45:19,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:22,406 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:23,052 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:45:24,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:29,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:30,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:45:30,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:45:30,806 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:45:32,480 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:34,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:39,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:42,553 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:44,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:45,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:45:45,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:45:45,802 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:45:49,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:45:52,624 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:45:53,054 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:45:55,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:00,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:00,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:46:00,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:46:00,804 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:46:02,693 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:05,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:10,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:12,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:15,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:46:15,802 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:46:15,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:46:16,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:21,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:22,847 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:23,053 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:46:26,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:30,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:46:30,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:46:30,892 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:46:31,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:34,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:36,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:41,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:44,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:45,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:46:45,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:46:45,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:46:47,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:52,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:46:52,136 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:46:52,136 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:46:52,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:46:52,138 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:46:53,054 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:46:53,086 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:46:54,091 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:46:58,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:00,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:47:00,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:47:00,893 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:47:04,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:04,181 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:09,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:14,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:14,247 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:15,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:47:15,891 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:47:15,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:47:20,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:23,054 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:47:24,327 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:26,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:31,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:31,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:47:31,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:47:31,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:47:34,391 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:36,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:41,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:44,455 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:46,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:47:46,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:47:46,154 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:47:47,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:53,054 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:47:53,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:47:54,539 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:47:58,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:01,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:48:01,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:48:01,155 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:48:03,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:04,624 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:08,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:13,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:14,688 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:16,152 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:48:16,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:48:16,153 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:48:18,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:23,055 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:48:24,056 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:24,758 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:29,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:31,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:48:31,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:48:31,625 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:48:34,829 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:34,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:39,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:44,897 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:45,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:46,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:48:46,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:48:46,622 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:48:50,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:48:53,056 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:48:54,975 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:48:56,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:01,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:02,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:49:02,092 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:49:02,093 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:49:05,061 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:49:06,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:11,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:15,147 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:49:17,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:49:17,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:17,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:49:17,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:49:22,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:23,057 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:49:24,225 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:49:27,776 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:32,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:49:32,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:49:32,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:49:33,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:38,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:40,322 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:49:43,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:47,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:49:47,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:49:47,389 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:49:48,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:50,393 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:49:53,058 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:49:54,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:49:59,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:00,467 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:02,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:50:02,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:50:02,436 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:50:04,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:09,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:12,547 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:14,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:17,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:50:17,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:50:17,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:50:20,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:21,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:50:21,166 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:50:21,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:50:21,170 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:50:21,604 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:50:22,608 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:23,058 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:50:25,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:26,627 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:30,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:32,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:50:32,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:50:32,391 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:50:34,692 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:36,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:41,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:44,762 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:46,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:47,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:50:47,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:50:47,388 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:50:51,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:50:53,059 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:50:56,841 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:50:56,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:01,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:03,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:51:03,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:51:03,177 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:51:06,905 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:07,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:12,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:16,969 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:18,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:51:18,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:51:18,173 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:51:18,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:23,066 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:51:24,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:27,041 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:29,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:33,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:51:33,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:51:33,196 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:51:35,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:37,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:40,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:46,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:47,174 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:48,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:51:48,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:51:48,194 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:51:51,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:51:53,066 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:51:57,247 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:51:57,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:02,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:03,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:52:03,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:52:03,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:52:07,318 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:52:07,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:12,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:16,385 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:52:18,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:18,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:52:18,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:52:18,485 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:52:23,067 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:52:23,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:26,450 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:52:28,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:33,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:33,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:52:33,921 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:52:33,923 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:52:36,519 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:52:39,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:44,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:46,594 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:52:48,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:52:48,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:52:48,918 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:52:50,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:53,069 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:52:55,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:52:56,685 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:00,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:04,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:53:04,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:53:04,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:53:05,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:06,766 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:10,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:16,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:16,825 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:19,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:53:19,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:53:19,278 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:53:21,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:23,070 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:53:26,888 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:27,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:32,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:34,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:53:34,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:53:34,279 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:53:36,967 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:37,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:42,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:44,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:53:44,467 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:53:44,468 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:53:44,471 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:53:45,021 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:53:47,035 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:53:48,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:49,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:53:49,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:53:49,278 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:53:53,071 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:53:54,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:53:57,111 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:00,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:04,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:54:04,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:54:04,279 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:54:05,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:07,178 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:10,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:15,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:17,266 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:19,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:54:19,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:54:19,278 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:54:20,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:23,072 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:54:25,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:27,340 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:30,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:34,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:54:34,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:54:34,592 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:54:35,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:37,413 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:40,932 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:46,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:47,490 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:54:49,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:54:49,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:54:49,588 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:54:51,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:54:53,072 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:54:57,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:02,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:04,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:55:04,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:55:04,587 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:55:07,836 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:12,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:17,838 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:19,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:55:19,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:55:19,588 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:55:23,073 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:55:23,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:28,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:32,712 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:55:33,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:34,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:55:34,588 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:55:34,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:55:38,798 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:42,784 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:55:44,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:49,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:55:51,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:55:51,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:55:51,038 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:55:53,074 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:55:54,861 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:55:55,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:00,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:04,929 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:05,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:06,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:56:06,036 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:56:06,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:56:11,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:15,006 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:16,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:21,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:56:21,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:56:21,735 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:56:21,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:23,074 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:56:25,075 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:27,076 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:32,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:35,156 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:36,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:56:36,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:56:36,733 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:56:37,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:42,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:45,235 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:48,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:53,076 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:56:53,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:56:53,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:56:53,380 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:56:54,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:56:57,337 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:56:59,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:04,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:07,408 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:57:08,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:57:08,378 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:57:08,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:57:09,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:14,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:17,470 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:57:20,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:23,077 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:57:24,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:57:24,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:57:24,653 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:57:25,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:27,542 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:57:30,915 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:36,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:37,607 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:57:39,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:57:39,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:57:39,650 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:57:41,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:46,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:49,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 17:57:49,338 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 17:57:49,445 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 17:57:49,517 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 17:57:50,083 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 17:57:51,129 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:57:52,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:57:53,078 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:57:54,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:57:54,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:57:54,650 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:57:57,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:02,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:07,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:09,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:58:09,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:58:09,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:58:13,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:18,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:23,089 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:58:24,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:24,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:58:24,676 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:58:24,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:58:29,920 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:33,506 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:58:35,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:39,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:58:39,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:58:39,658 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:58:40,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:46,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:47,722 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:58:51,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:58:53,079 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:58:54,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:58:54,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:58:54,669 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:58:56,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:01,073 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:59:01,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:06,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:10,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:59:10,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:59:10,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:59:12,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:13,180 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:59:17,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:22,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:23,080 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:59:25,351 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:59:25,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:59:25,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:59:25,814 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:59:28,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:33,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:35,428 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:59:38,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:40,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:59:40,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:59:40,814 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:59:44,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:47,513 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 17:59:49,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:53,083 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 17:59:55,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 17:59:55,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 17:59:55,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 17:59:55,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 17:59:57,577 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:00,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:05,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:09,651 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:10,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:00:10,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:00:10,815 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:00:11,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:16,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:19,731 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:22,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:23,083 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:00:25,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:00:25,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:00:25,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:00:27,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:31,830 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:32,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:37,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:40,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:00:40,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:00:40,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:00:41,952 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:43,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:48,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:52,028 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:00:53,083 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:00:53,083 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:00:55,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:00:55,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:00:55,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:00:59,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:03,119 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:01:04,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:09,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:11,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:01:11,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:01:11,404 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:01:15,026 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:15,195 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:01:20,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:23,087 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:01:25,085 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:25,307 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:01:26,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:01:26,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:01:26,403 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:01:30,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:35,373 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:01:36,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:41,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:01:41,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:01:41,402 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:01:41,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:47,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:47,473 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:01:52,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:53,087 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:01:56,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:01:56,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:01:56,406 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:01:57,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:01:59,564 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:02,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:07,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:08,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:02:08,531 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:02:08,532 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:02:08,536 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:02:08,629 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:02:09,642 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:11,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:02:11,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:02:11,452 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:02:12,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:17,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:21,752 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:23,088 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:02:23,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:26,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:02:26,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:02:26,436 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:02:28,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:31,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:33,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:38,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:41,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:02:41,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:02:41,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:02:43,955 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:44,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:49,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:53,089 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:02:54,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:02:55,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:02:56,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:02:56,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:02:56,435 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:03:00,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:04,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:03:06,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:11,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:12,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:03:12,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:03:12,018 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:03:16,195 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:03:17,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:22,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:23,090 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:03:25,279 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:03:27,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:03:27,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:03:27,015 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:03:28,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:33,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:35,357 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:03:38,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:42,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:03:42,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:03:42,018 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:03:44,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:47,469 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:03:49,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:53,092 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:03:55,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:03:57,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:03:57,038 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:03:57,060 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:03:57,544 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:00,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:05,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:09,895 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:11,086 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:12,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:04:12,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:04:12,017 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:04:16,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:19,969 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:21,721 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:23,092 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:04:27,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:04:27,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:04:27,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:27,022 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:04:30,059 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:32,336 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:37,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:42,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:04:42,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:04:42,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:04:42,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:43,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:48,261 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:52,250 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:04:53,093 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:04:54,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:04:57,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:04:57,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:04:57,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:04:59,243 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:02,331 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:05,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:10,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:12,016 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:05:12,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:05:12,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:05:14,433 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:15,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:20,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:23,094 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:05:23,503 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:26,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:27,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:05:27,020 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:05:27,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:05:31,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:35,587 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:36,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:41,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:42,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:05:42,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:05:42,639 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:05:43,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:05:43,919 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:05:43,919 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:05:43,929 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:05:44,647 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:05:45,652 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:46,933 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:51,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:53,096 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:05:55,749 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:05:57,595 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:05:57,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:05:57,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:05:57,635 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:06:02,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:07,844 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:06:08,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:12,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:06:12,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:06:12,636 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:06:13,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:19,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:19,937 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:06:23,095 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:06:25,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:27,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:06:27,635 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:06:27,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:06:30,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:35,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:36,045 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:06:40,909 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:43,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:06:43,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:06:43,252 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:06:46,128 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:06:46,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:51,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:53,099 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:06:56,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:06:58,202 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:06:58,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:06:58,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:06:58,249 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:07:02,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:08,273 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:07:08,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:13,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:07:13,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:07:13,249 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:07:13,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:18,343 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:07:18,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:23,097 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:07:24,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:28,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:07:28,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:07:28,251 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:07:29,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:30,437 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:07:34,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:39,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:40,513 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:07:43,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:07:43,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:07:43,250 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:07:45,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:50,589 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:07:51,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:53,097 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:07:57,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:07:58,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:07:58,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:07:58,252 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:08:01,685 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:02,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:07,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:11,771 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:13,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:08:13,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:08:13,250 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:08:18,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:21,845 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:23,098 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:08:24,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:29,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:29,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:08:29,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:08:29,579 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:08:32,005 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:34,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:39,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:42,084 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:44,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:08:44,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:08:44,575 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:08:45,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:50,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:53,099 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:08:54,176 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:08:56,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:08:59,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:08:59,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:08:59,578 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:09:01,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:04,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:07,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:12,781 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:14,322 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:14,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:09:14,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:09:14,576 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:09:17,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:22,838 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:23,100 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:09:23,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:09:23,407 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:09:23,407 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:09:23,409 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:09:24,405 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:24,406 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:09:28,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:29,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:09:29,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:09:29,576 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:09:33,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:36,487 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:39,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:44,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:44,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:09:44,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:09:44,578 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:09:46,552 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:49,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:53,103 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:09:55,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:09:56,621 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:09:59,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:09:59,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:09:59,577 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:10:00,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:06,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:08,710 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:10:11,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:14,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:10:14,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:10:14,720 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:10:16,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:18,786 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:10:21,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:23,102 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:10:27,901 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:10:27,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:29,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:10:29,729 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:10:29,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:10:32,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:38,007 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:10:38,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:43,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:44,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:10:44,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:10:44,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:10:48,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:50,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:10:53,113 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:10:54,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:59,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:10:59,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:10:59,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:10:59,761 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:11:00,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:04,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:10,290 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:10,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:14,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:11:14,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:11:14,762 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:11:15,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:20,364 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:21,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:23,103 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:11:27,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:29,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:11:29,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:11:29,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:11:32,485 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:32,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:37,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:42,562 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:43,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:44,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:11:44,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:11:44,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:11:49,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:52,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:11:53,104 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:11:54,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:11:59,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:00,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:12:00,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:12:00,424 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:12:04,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:04,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:09,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:14,778 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:15,094 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:15,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:12:15,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:12:15,424 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:12:20,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:23,104 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:12:24,855 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:26,109 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:31,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:31,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:12:31,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:12:31,530 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:12:34,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:36,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:41,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:46,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:12:46,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:12:46,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:12:47,034 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:47,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:52,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:12:53,107 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:12:55,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:12:55,186 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:12:55,187 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:12:55,189 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:12:56,101 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:12:56,102 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:12:58,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:01,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:13:01,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:13:01,530 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:13:03,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:08,181 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:13:09,011 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:14,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:16,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:13:16,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:13:16,527 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:13:18,260 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:13:19,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:23,113 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:13:25,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:28,353 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:13:30,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:31,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:13:31,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:13:31,528 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:13:35,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:38,427 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:13:41,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:46,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:13:46,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:13:46,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:13:46,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:50,517 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:13:51,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:13:53,114 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:13:57,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:00,580 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:01,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:14:01,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:14:01,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:14:02,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:07,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:10,648 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:13,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:17,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:14:17,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:14:17,318 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:14:18,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:20,719 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:23,114 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:14:24,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:30,018 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:30,803 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:32,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:14:32,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:14:32,316 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:14:35,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:40,744 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:40,884 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:45,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:47,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:14:47,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:14:47,317 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:14:50,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:14:52,981 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:14:53,115 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:14:56,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:01,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:02,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:15:02,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:15:02,317 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:15:03,060 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:15:06,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:11,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:13,124 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:15:16,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:17,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:15:17,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:15:17,317 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:15:22,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:23,116 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:15:25,208 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:15:28,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:32,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:15:32,318 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:15:32,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:15:33,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:35,275 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:15:38,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:44,118 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:44,341 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:15:47,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:15:47,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:15:47,318 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:15:49,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:53,117 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:15:55,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:15:56,536 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:00,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:03,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:16:03,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:16:03,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:16:05,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:06,612 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:10,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:16,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:16,686 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:18,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:16:18,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:16:18,204 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:16:21,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:23,119 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:16:25,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:16:25,429 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:16:25,429 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:16:25,432 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:16:25,759 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:16:26,768 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:27,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:32,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:33,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:16:33,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:16:33,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:16:38,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:38,857 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:43,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:48,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:16:48,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:48,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:16:48,206 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:16:48,948 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:53,119 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:16:54,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:16:59,030 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:16:59,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:03,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:17:03,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:17:03,205 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:17:04,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:09,128 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:17:10,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:15,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:18,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:17:18,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:17:18,205 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:17:19,197 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:17:21,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:23,119 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:17:27,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:31,277 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:17:32,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:33,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:17:33,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:17:33,205 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:17:38,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:41,400 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:17:43,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:48,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:48,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:17:48,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:17:48,649 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:17:51,460 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:17:53,120 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:17:54,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:17:59,123 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:02,544 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:18:03,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:18:03,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:18:03,646 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:18:04,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:09,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:12,634 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:18:15,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:18,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:18:18,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:18:18,647 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:18:20,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:23,122 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:18:24,743 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:18:26,123 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:31,125 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:33,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:18:33,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:18:33,649 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:18:34,813 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:18:36,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:41,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:46,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:48,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:18:48,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:18:48,647 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:18:48,937 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:18:52,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:18:53,123 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:18:58,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:01,041 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:19:03,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:03,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:19:03,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:19:03,647 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:19:08,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:11,114 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:19:14,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:18,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:19:18,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:19:18,651 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:19:19,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:23,147 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:19:25,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:25,200 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:19:30,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:33,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:19:33,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:19:33,648 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:19:35,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:37,296 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:19:40,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:45,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:47,392 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:19:48,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:19:48,647 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:19:48,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:19:50,870 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:53,125 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:19:56,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:19:59,483 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:01,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:03,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:20:03,648 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:20:03,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:20:06,915 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:07,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:20:07,667 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:20:07,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:20:07,669 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:20:08,607 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:08,608 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:20:09,613 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:12,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:17,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:18,648 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:20:18,649 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:20:18,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:20:20,743 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:22,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:23,126 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:20:28,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:30,814 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:33,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:33,647 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:20:33,650 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:20:33,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:20:38,859 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:40,902 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:44,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:49,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:20:49,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:20:49,031 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:20:49,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:20:50,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:20:53,141 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:20:55,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:00,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:03,155 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:21:04,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:21:04,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:21:04,028 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:21:05,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:10,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:15,347 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:21:15,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:19,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:21:19,029 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:21:19,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:21:21,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:23,127 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:21:25,418 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:21:26,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:31,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:34,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:21:34,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:21:34,031 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:21:36,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:37,522 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:21:41,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:46,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:47,614 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:21:49,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:21:49,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:21:49,029 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:21:52,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:53,129 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:21:57,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:21:59,709 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:22:02,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:04,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:22:04,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:22:04,032 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:22:08,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:09,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:22:13,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:18,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:19,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:22:19,031 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:22:19,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:22:20,889 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:22:23,130 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:22:24,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:29,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:34,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:22:34,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:22:34,035 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:22:34,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:37,036 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:22:39,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:44,796 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:47,106 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:22:49,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:22:49,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:22:49,033 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:22:50,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:53,132 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:22:56,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:22:59,220 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:01,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:04,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:23:04,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:23:04,031 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:23:06,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:09,297 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:11,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:16,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:19,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:23:19,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:23:19,033 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:23:21,390 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:21,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:23,131 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:23:27,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:32,975 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:33,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:34,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:23:34,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:23:34,031 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:23:38,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:43,548 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:43,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:48,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:49,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:23:49,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:23:49,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:23:52,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:23:52,547 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:23:52,552 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:23:52,558 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:23:52,623 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:23:53,132 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:23:53,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:54,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:23:54,636 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:23:59,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:04,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:24:04,034 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:24:04,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:24:04,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:05,719 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:24:09,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:14,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:15,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:24:19,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:24:19,032 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:24:19,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:24:19,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:23,134 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:24:25,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:27,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:24:30,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:34,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:24:34,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:24:34,607 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:24:35,846 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:37,209 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:24:40,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:45,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:49,289 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:24:49,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:24:49,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:24:49,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:24:51,784 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:53,134 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:24:57,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:24:59,368 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:02,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:04,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:25:04,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:25:04,605 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:25:07,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:09,449 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:13,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:18,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:19,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:25:19,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:25:19,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:25:21,544 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:23,135 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:25:24,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:29,137 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:31,626 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:34,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:34,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:25:34,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:25:34,605 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:25:39,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:41,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:45,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:49,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:25:49,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:25:49,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:25:51,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:25:53,135 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:25:53,803 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:25:57,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:02,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:03,921 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:04,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:26:04,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:26:04,975 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:26:07,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:12,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:16,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:17,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:19,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:26:19,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:26:19,972 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:26:23,137 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:26:23,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:26,094 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:28,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:33,264 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:34,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:26:34,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:26:34,974 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:26:38,190 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:38,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:43,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:47,252 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:48,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:49,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:26:49,972 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:26:49,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:26:53,140 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:26:54,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:26:57,330 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:26:59,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:04,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:05,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:27:05,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:27:05,631 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:27:07,406 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:09,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:14,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:19,499 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:20,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:20,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:27:20,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:27:20,627 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:27:23,140 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:27:26,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:28,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:27:28,877 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:27:28,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:27:28,880 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:27:29,575 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:29,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:27:31,589 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:31,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:35,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:27:35,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:27:35,628 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:27:37,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:41,668 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:43,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:48,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:50,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:27:50,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:27:50,630 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:27:51,740 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:27:53,140 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:27:54,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:27:59,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:03,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:04,858 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:05,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:28:05,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:28:05,628 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:28:10,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:13,928 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:16,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:21,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:21,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:28:21,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:28:21,849 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:28:23,141 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:28:26,037 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:27,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:32,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:36,119 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:36,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:28:36,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:28:36,849 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:28:38,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:43,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:48,210 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:48,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:51,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:28:51,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:28:51,850 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:28:53,145 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:28:54,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:28:58,288 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:28:59,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:04,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:06,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:29:06,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:29:06,852 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:29:08,355 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:29:10,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:15,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:19,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:29:20,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:21,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:29:21,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:29:21,851 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:29:23,144 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:29:26,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:31,550 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:31,610 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:29:36,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:36,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:29:36,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:29:36,851 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:29:41,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:43,704 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:29:46,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:51,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:29:51,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:29:51,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:29:51,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:29:53,144 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:29:53,791 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:29:57,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:02,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:05,869 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:30:06,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:30:06,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:30:06,852 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:30:08,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:13,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:15,952 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:30:18,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:21,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:30:21,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:30:21,852 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:30:23,146 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:30:24,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:28,076 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:30:30,014 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:35,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:36,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:30:36,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:30:36,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:30:40,298 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:30:40,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:45,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:50,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:30:51,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:30:51,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:30:51,853 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:30:52,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:30:53,146 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:30:56,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:01,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:02,478 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:06,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:06,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:31:06,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:31:06,853 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:31:12,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:13,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:31:13,290 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:31:13,291 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:31:13,293 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:31:13,562 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:31:14,570 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:15,577 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:17,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:22,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:23,148 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:31:23,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:31:23,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:31:23,236 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:31:25,650 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:27,872 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:32,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:37,751 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:38,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:38,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:31:38,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:31:38,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:31:43,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:47,869 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:31:49,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:53,148 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:31:53,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:31:53,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:31:53,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:31:54,452 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:31:59,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:00,101 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:04,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:08,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:32:08,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:32:08,516 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:32:10,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:12,187 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:15,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:21,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:22,308 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:23,149 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:32:23,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:32:23,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:32:23,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:32:26,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:32,083 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:32,384 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:37,084 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:38,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:32:38,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:32:38,517 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:32:42,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:44,471 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:47,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:52,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:32:53,150 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:32:53,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:32:53,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:32:53,514 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:32:54,584 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:32:58,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:03,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:06,672 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:33:08,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:33:08,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:33:08,513 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:33:08,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:13,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:16,742 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:33:19,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:23,152 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:33:23,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:33:23,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:33:23,517 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:33:24,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:26,820 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:33:29,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:34,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:35,892 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:33:38,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:33:38,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:33:38,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:33:39,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:44,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:45,962 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:33:50,620 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:53,152 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:33:53,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:33:53,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:33:53,517 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:33:55,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:33:58,136 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:00,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:05,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:08,212 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:08,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:34:08,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:34:08,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:34:11,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:16,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:18,282 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:22,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:23,153 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:34:23,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:34:23,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:34:23,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:34:27,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:30,365 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:33,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:38,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:38,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:34:38,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:34:38,520 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:34:40,435 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:44,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:49,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:50,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:34:50,212 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:34:50,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:34:50,215 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:34:50,500 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:34:51,507 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:52,513 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:34:53,154 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:34:53,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:34:53,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:34:53,516 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:34:54,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:34:59,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:02,587 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:35:04,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:09,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:09,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:35:09,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:35:09,988 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:35:12,664 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:35:15,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:20,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:23,155 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:35:24,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:35:24,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:35:24,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:35:24,986 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:35:26,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:31,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:34,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:35:36,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:39,984 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:35:39,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:35:39,985 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:35:42,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:47,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:52,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:35:53,156 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:35:54,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:35:54,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:35:54,988 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:35:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:00,200 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:03,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:08,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:09,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:36:09,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:36:09,986 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:36:12,296 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:14,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:19,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:22,356 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:23,169 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:36:24,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:36:24,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:36:24,986 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:36:25,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:30,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:32,433 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:35,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:39,986 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:36:39,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:36:39,989 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:36:41,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:44,516 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:46,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:51,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:36:53,158 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:36:54,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:36:54,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:36:54,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:36:56,618 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:36:57,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:02,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:07,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:08,754 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:37:09,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:37:09,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:37:09,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:37:13,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:19,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:20,845 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:37:23,158 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:37:24,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:37:24,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:37:24,993 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:37:25,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:30,940 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:37:31,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:36,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:39,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:37:39,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:37:39,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:37:41,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:43,033 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:37:46,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:51,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:37:53,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:37:53,159 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:37:54,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:37:54,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:37:54,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:37:57,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:02,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:03,180 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:07,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:09,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:38:09,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:38:09,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:38:13,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:14,285 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:18,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:23,161 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:38:24,163 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:24,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:38:24,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:38:24,994 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:38:26,392 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:29,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:34,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:36,471 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:39,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:39,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:38:39,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:38:39,995 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:38:45,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:46,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:38:46,157 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:38:46,158 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:38:46,161 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:38:46,553 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:46,553 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:38:48,561 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:38:51,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:53,161 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:38:55,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:38:55,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:38:55,920 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:38:57,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:38:58,668 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:39:02,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:07,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:11,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:39:11,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:39:11,644 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:39:12,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:14,790 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:39:18,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:23,163 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:39:24,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:26,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:39:26,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:39:26,641 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:39:26,881 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:39:29,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:34,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:37,065 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:39:40,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:41,640 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:39:41,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:39:41,641 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:39:45,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:49,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:39:50,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:53,163 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:39:56,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:39:56,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:39:56,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:39:56,644 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:39:59,233 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:01,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:06,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:09,306 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:11,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:40:11,642 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:40:11,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:40:11,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:16,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:21,401 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:22,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:23,164 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:40:26,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:40:26,642 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:40:26,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:40:27,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:32,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:32,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:37,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:41,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:40:41,642 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:40:41,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:40:42,558 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:42,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:47,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:53,164 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:40:53,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:40:54,676 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:40:56,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:40:56,642 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:40:56,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:40:58,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:04,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:04,757 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:41:09,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:11,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:41:11,643 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:41:11,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-22 18:41:14,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:16,859 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:41:19,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:23,165 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:41:25,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:26,641 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:41:26,642 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:41:26,949 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:41:30,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:36,840 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:37,077 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:41:41,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:41:41,643 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:41:41,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:47,702 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:49,315 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:41:52,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:53,166 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:41:56,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:41:56,646 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:41:58,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:41:59,399 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:03,616 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:09,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:09,602 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:11,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:42:11,643 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:42:14,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:20,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:20,677 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:23,166 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:42:26,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:26,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:42:26,646 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:42:30,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:42:30,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:42:30,511 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:42:30,517 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:42:30,925 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:30,926 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:42:31,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:31,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:36,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:41,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:41,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:42:41,645 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:42:43,009 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:46,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:52,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:42:53,076 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:42:53,166 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:42:56,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:42:56,645 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:42:57,911 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:03,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:05,164 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:43:08,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:12,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:43:12,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:43:12,996 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:43:14,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:15,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:43:19,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:23,169 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:43:25,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:27,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:43:27,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:43:27,990 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:43:30,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:35,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:37,436 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:43:40,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:42,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:43:42,990 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:43:46,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:49,583 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:43:51,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:53,169 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:43:57,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:43:57,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:43:57,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:43:57,992 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:43:59,663 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:02,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:07,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:11,750 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:12,989 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:44:12,990 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:44:13,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:18,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:20,819 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:23,169 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:44:24,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:29,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:44:29,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:44:29,049 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:44:29,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:30,901 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:34,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:39,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:42,991 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:44,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:44:44,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:44:44,047 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:44:45,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:50,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:53,069 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:44:53,170 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:44:56,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:44:59,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:44:59,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:44:59,047 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:45:01,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:05,165 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:45:06,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:11,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:14,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:45:14,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:45:14,049 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:45:15,272 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:45:17,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:22,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:23,172 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:45:27,372 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:45:27,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:29,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:45:29,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:45:29,047 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:45:33,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:37,465 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:45:38,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:43,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:44,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:45:44,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:45:44,051 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:45:49,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:49,555 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:45:53,172 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:45:55,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:45:59,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:45:59,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:45:59,050 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:45:59,628 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:46:00,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:05,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:09,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:46:09,487 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:46:09,487 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:46:09,490 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:46:09,699 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:46:10,703 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:46:10,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:11,709 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:46:13,717 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:46:14,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:46:14,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:46:14,049 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:46:16,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:21,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:23,183 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:46:27,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:29,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:46:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:46:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:46:32,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:37,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:42,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:44,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:46:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:46:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:46:47,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:52,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:53,173 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:46:58,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:46:59,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:46:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:46:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:47:03,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:08,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:13,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:14,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:47:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:47:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:47:19,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:23,173 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:47:25,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:29,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:47:29,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:47:29,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:47:30,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:35,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:40,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:44,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:47:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:47:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:47:46,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:51,286 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:53,174 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:47:57,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:47:59,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:47:59,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:47:59,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:48:02,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:07,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:12,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:14,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:48:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:48:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:48:18,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:23,177 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:48:24,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:29,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:48:29,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:48:29,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:48:29,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:34,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:39,284 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:44,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:48:44,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:48:44,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:48:44,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:49,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:53,175 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:48:55,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:48:59,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:48:59,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:48:59,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:49:00,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:05,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:10,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:14,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:49:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:49:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:49:16,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:21,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:23,176 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:49:27,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:29,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:49:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:49:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:49:32,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:37,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:42,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:44,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:49:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:49:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:49:48,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:53,176 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:49:54,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:49:59,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:49:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:49:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:49:59,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:04,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:09,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:14,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:50:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:50:14,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:50:14,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:19,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:23,177 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:50:25,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:29,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:50:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:50:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:50:30,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:35,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:40,352 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:44,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:50:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:50:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:50:46,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:51,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:53,177 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:50:57,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:50:59,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:50:59,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:50:59,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:51:02,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:07,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:12,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:14,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:51:14,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:51:14,106 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:51:17,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:22,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:23,178 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:51:28,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:29,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:51:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:51:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:51:33,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:38,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:43,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:44,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:51:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:51:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:51:49,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:53,178 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:51:55,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:51:59,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:51:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:51:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:52:00,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:05,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:10,268 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:14,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:52:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:52:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:52:15,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:20,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:23,179 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:52:26,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:29,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:52:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:52:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:52:31,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:36,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:41,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:44,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:52:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:52:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:52:46,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:51,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:53,179 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:52:57,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:52:59,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:52:59,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:52:59,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:53:02,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:07,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:12,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:14,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:53:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:53:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:53:17,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:22,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:23,180 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:53:28,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:29,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:53:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:53:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:53:33,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:38,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:43,297 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:44,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:53:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:53:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:53:49,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:53,180 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:53:55,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:53:59,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:53:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:53:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:54:00,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:05,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:10,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:14,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:54:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:54:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:54:15,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:20,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:23,181 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:54:26,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:29,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:54:29,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:54:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:54:31,359 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:36,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:41,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:44,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:54:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:54:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:54:47,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:52,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:53,182 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:54:58,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:54:59,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:54:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:54:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:55:03,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:08,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:13,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:14,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:55:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:55:14,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:55:19,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:23,182 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:55:25,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:29,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:55:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:55:29,103 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:55:30,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:35,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:40,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:44,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:55:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:55:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:55:46,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:51,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:53,183 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:55:57,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:55:59,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:55:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:55:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:56:02,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:07,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:12,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:14,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:56:14,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:56:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:56:18,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:23,183 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:56:24,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:29,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:56:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:56:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:56:29,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:34,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:39,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:44,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:56:44,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:56:44,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:56:44,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:49,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:53,184 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:56:55,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:56:59,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:56:59,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:56:59,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:57:00,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:05,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:10,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:14,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:57:14,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:57:14,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:57:15,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:20,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:23,185 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:57:26,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:29,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:57:29,095 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:57:29,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:57:31,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:36,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:39,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 18:57:39,452 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 18:57:39,453 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 18:57:39,454 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 18:57:39,454 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 18:57:39,455 DEBUG SenderThread:2219461 [sender.py:send():380] send: metric 2023-11-22 18:57:39,455 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 18:57:39,455 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 18:57:39,458 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 18:57:40,347 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:57:40,348 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 18:57:41,353 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:57:41,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:42,362 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/config.yaml 2023-11-22 18:57:44,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:57:44,383 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:57:45,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:57:45,011 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:57:48,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:53,186 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:57:54,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:59,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:57:59,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:58:00,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:58:00,915 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:58:05,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:10,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:14,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:58:14,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:58:14,717 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:58:15,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:16,605 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:58:19,621 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:58:21,954 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:23,186 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:58:27,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:32,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:37,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:39,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:58:39,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:58:39,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:58:41,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:58:42,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:47,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:47,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:58:52,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:58:53,189 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:58:58,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:03,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:59:03,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:59:03,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:04,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 18:59:08,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:10,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:59:10,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:59:10,441 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:59:13,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:18,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:23,189 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:59:24,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:25,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:59:25,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:59:25,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:59:29,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:34,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:39,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:40,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:59:40,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:59:40,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:59:45,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:50,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 18:59:53,194 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 18:59:55,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 18:59:55,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 18:59:55,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 18:59:55,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:00,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:05,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:10,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:00:10,485 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:00:10,486 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:00:11,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:16,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:20,310 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:00:22,017 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:23,206 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:00:25,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:00:25,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:00:25,441 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:00:27,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:33,236 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:34,463 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:00:38,237 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:40,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:00:40,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:00:40,456 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:00:43,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:46,592 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:00:48,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:53,210 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:00:54,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:00:55,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:00:55,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:00:55,442 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:01:00,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:00,684 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:05,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:10,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:10,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:01:10,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:01:10,442 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:01:12,771 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:16,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:21,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:23,208 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:01:24,862 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:25,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:01:25,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:01:25,494 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:01:26,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:31,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:34,950 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:36,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:40,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:01:40,493 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:01:42,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:47,042 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:48,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:53,209 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:01:54,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:01:55,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:01:55,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:01:55,494 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:01:57,132 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:01:59,732 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:04,734 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:08,219 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:02:10,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:10,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:02:10,493 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:02:10,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:02:15,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:18,284 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:02:20,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:23,209 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:02:26,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:27,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:02:27,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:02:27,049 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:02:30,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:02:31,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:36,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:40,436 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:02:41,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:42,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:02:42,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:02:42,047 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:02:46,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:50,532 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:02:52,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:02:53,210 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:02:57,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:02:57,048 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:02:57,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:02,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:02,630 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:07,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:12,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:03:12,047 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:03:13,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:14,718 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:18,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:23,210 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:03:24,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:24,800 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:27,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:03:27,048 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:03:29,243 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:34,872 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:34,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:39,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:43,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:03:43,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:03:43,656 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:03:45,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:46,978 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:50,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:53,212 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:03:56,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:03:57,093 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:03:58,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:03:58,652 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:04:01,871 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:06,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:07,015 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:04:07,017 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:04:07,018 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:04:07,021 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:04:07,168 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:04:08,179 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:09,184 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:12,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:13,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:04:13,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:04:13,653 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:04:17,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:21,307 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:23,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:23,213 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:04:28,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:28,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:04:28,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:04:28,926 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:04:30,377 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:34,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:39,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:42,487 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:43,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:04:43,943 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:04:45,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:50,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:53,213 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:04:54,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:04:55,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:04:58,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:04:58,943 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:05:01,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:06,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:06,828 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:05:11,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:15,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:05:15,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:05:15,556 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:05:16,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:19,121 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:05:22,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:23,214 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:05:27,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:30,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:05:30,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:05:32,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:37,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:41,270 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:05:42,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:45,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:05:45,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:05:45,557 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:05:48,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:53,214 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:05:53,351 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:05:54,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:05:59,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:00,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:06:00,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:06:00,561 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:06:03,429 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:06:04,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:09,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:12,526 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:06:15,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:15,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:06:15,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:06:20,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:23,221 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:06:24,610 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:06:26,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:30,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:06:30,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:06:31,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:36,695 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:06:37,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:42,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:45,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:06:45,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:06:45,557 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:06:46,777 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:06:47,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:52,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:53,215 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:06:58,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:06:58,840 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:00,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:07:00,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:07:03,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:09,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:10,925 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:14,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:15,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:07:15,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:07:19,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:21,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:23,216 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:07:25,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:30,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:30,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:07:30,557 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:07:33,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:36,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:41,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:45,182 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:45,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:07:45,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:07:46,798 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:51,800 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:07:53,218 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:07:55,370 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:07:56,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:00,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:08:00,556 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:08:02,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:05,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:08:05,888 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:08:05,894 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:08:05,898 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:08:06,466 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:08:07,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:07,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:08,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:12,901 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:16,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:08:16,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:08:16,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:08:18,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:19,658 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:23,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:23,218 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:08:28,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:30,746 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:31,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:08:31,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:08:31,479 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:08:33,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:38,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:42,871 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:44,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:46,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:08:46,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:08:49,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:08:53,218 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:08:54,968 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:08:55,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:00,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:02,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:09:02,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:09:02,592 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:09:05,047 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:09:05,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:10,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:15,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:17,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:09:17,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:09:17,590 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:09:20,778 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:23,224 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:09:26,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:29,235 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:09:32,082 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:32,590 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:09:32,591 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:09:37,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:41,335 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:09:43,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:48,005 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:48,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:09:48,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:09:48,929 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:09:53,221 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:09:53,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:09:53,523 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:09:58,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:03,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:03,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:10:03,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:10:03,932 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:10:08,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:09,630 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:10:13,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:18,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:18,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:10:18,931 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:10:21,722 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:10:23,221 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:10:24,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:29,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:31,846 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:10:33,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:10:33,932 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:10:35,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:40,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:43,017 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:10:45,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:48,932 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:10:48,932 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:10:48,933 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:10:51,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:10:53,224 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:10:55,121 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:10:56,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:01,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:04,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:11:04,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:11:04,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:11:06,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:07,217 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:11:11,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:16,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:19,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:11:19,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:11:19,174 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:11:19,323 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:11:22,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:23,224 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:11:28,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:31,013 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:11:34,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:34,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:11:34,172 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:11:39,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:41,089 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:11:44,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:49,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:11:49,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:11:49,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:11:53,218 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:11:53,225 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:11:55,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:00,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:03,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:12:03,300 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:12:03,301 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:12:03,304 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:12:04,305 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:04,307 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:12:04,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:12:04,737 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:12:05,311 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:05,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:10,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:16,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:17,468 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:19,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:12:19,737 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:12:21,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:23,226 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:12:27,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:27,574 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:32,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:35,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:12:35,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:12:38,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:39,682 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:43,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:48,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:49,772 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:12:50,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:12:50,397 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:12:53,226 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:12:53,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:12:58,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:01,855 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:03,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:05,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:13:05,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:13:08,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:11,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:14,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:19,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:20,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:13:20,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:13:20,401 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:13:22,000 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:23,227 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:13:25,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:30,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:34,075 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:35,397 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:13:35,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:13:35,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:40,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:44,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:46,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:50,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:13:50,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:13:51,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:13:53,205 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:13:53,227 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:13:56,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:01,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:03,278 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:14:05,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:14:05,398 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:14:06,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:11,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:13,342 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:14:17,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:21,110 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:14:21,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:14:21,113 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:14:22,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:23,228 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:14:23,420 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:14:28,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:33,498 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:14:33,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:36,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:14:36,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:14:36,112 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:14:39,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:44,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:45,579 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:14:49,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:51,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:14:51,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:14:51,826 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:14:53,229 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:14:55,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:14:55,646 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:00,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:05,715 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:06,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:06,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:15:06,824 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:15:12,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:15,795 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:17,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:22,927 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:22,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:15:22,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:15:22,961 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:15:23,230 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:15:25,868 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:28,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:33,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:34,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:15:34,844 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:15:34,845 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:15:34,848 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:15:34,942 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:15:36,952 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:37,957 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:37,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:15:37,958 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:15:39,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:44,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:48,037 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:49,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:53,231 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:15:54,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:15:54,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:15:54,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:15:54,277 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:15:58,106 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:15:59,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:04,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:08,180 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:16:09,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:16:09,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:16:09,275 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:16:10,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:15,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:20,255 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:16:21,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:23,231 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:16:24,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:16:24,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:16:24,279 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:16:26,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:30,331 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:16:32,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:37,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:39,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:16:39,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:16:39,276 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:16:40,400 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:16:42,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:47,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:49,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:16:52,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:53,232 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:16:54,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:16:54,276 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:16:58,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:16:59,547 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:03,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:08,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:09,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:17:09,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:17:09,276 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:17:09,632 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:13,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:18,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:21,720 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:23,232 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:17:24,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:24,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:17:24,277 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:17:29,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:31,783 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:34,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:39,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:17:39,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:17:39,277 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:17:39,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:41,860 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:44,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:49,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:17:51,947 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:17:53,233 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:17:54,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:17:54,277 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:17:55,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:00,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:02,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:05,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:09,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:18:09,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:18:09,661 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:18:10,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:12,073 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:15,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:20,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:23,234 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:18:24,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:24,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:18:24,660 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:18:25,910 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:30,912 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:34,227 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:36,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:40,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:18:40,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:18:40,261 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:18:41,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:44,303 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:46,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:51,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:18:53,234 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:18:54,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:18:55,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:18:55,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:18:55,259 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:18:56,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:01,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:04,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:19:04,273 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:19:04,274 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:19:04,277 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:19:04,460 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:19:06,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:07,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:07,478 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:10,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:19:10,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:19:10,261 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:19:12,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:16,539 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:18,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:23,035 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:23,236 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:19:25,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:19:25,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:19:25,261 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:19:28,665 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:28,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:33,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:37,740 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:39,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:40,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:19:40,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:19:40,260 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:19:45,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:47,817 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:19:51,037 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:53,236 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:19:56,237 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:19:56,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:19:56,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:19:56,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:19:59,901 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:01,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:06,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:09,964 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:11,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:20:11,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:20:11,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:20:12,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:17,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:20,034 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:23,237 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:20:23,237 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:27,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:20:27,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:20:27,509 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:20:28,783 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:30,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:33,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:38,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:40,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:42,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:20:42,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:20:42,507 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:20:44,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:50,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:52,257 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:20:53,237 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:20:56,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:20:58,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:20:58,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:20:58,268 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:21:01,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:02,333 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:06,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:11,768 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:12,398 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:13,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:21:13,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:21:13,267 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:21:17,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:22,464 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:23,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:23,246 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:21:28,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:29,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:21:29,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:21:29,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:21:32,543 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:34,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:39,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:44,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:44,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:21:44,431 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:21:44,626 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:49,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:21:53,246 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:21:54,686 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:21:55,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:00,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:00,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:22:00,810 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:22:00,811 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:22:04,748 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:06,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:11,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:14,819 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:15,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:22:15,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:22:15,805 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:22:17,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:22,022 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:23,247 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:22:24,915 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:27,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:31,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:22:31,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:22:31,280 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:22:32,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:32,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:22:32,667 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:22:32,668 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:22:32,670 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:22:32,975 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:22:33,980 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:34,985 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:37,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:42,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:44,055 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:46,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:22:46,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:22:46,280 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:22:48,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:53,248 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:22:53,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:22:56,236 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:22:58,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:02,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:23:02,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:23:02,851 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:23:04,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:06,294 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:23:09,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:14,224 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:16,353 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:23:17,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:23:17,847 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:23:20,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:23,250 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:23:26,027 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:26,429 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:23:31,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:34,009 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:23:34,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:23:34,012 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:23:36,209 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:36,499 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:23:41,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:46,567 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:23:47,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:49,010 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:23:49,010 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:23:52,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:53,251 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:23:57,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:23:58,652 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:02,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:04,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:24:04,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:24:04,594 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:24:07,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:08,726 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:12,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:18,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:18,793 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:19,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:24:19,592 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:24:23,253 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:24:24,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:28,868 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:30,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:35,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:35,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:24:35,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:24:35,328 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:24:38,938 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:40,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:45,625 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:49,019 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:24:50,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:24:50,325 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:24:51,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:53,253 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:24:57,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:24:59,104 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:02,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:06,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:25:06,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:25:06,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:25:07,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:08,177 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:12,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:17,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:20,245 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:21,132 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:25:21,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:25:23,253 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:25:23,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:28,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:33,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:34,323 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:36,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:25:36,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:25:38,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:43,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:46,401 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:48,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:51,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:25:51,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:25:53,255 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:25:54,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:25:56,471 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:25:59,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:04,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:05,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:26:05,305 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:26:05,305 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:26:05,308 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:26:05,528 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:26:06,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:26:06,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:26:06,533 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:07,538 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:10,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:15,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:16,596 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:20,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:21,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:26:21,134 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:26:23,260 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:26:25,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:26,667 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:30,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:35,646 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:36,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:26:36,133 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:26:36,740 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:41,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:46,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:48,816 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:26:51,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:26:51,134 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:26:52,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:53,258 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:26:58,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:26:58,885 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:03,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:06,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:27:06,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:27:08,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:08,953 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:13,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:18,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:19,043 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:21,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:27:21,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:27:23,257 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:27:24,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:29,108 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:29,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:34,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:36,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:27:36,137 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:27:39,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:40,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:45,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:49,285 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:27:50,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:51,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:27:51,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:27:53,258 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:27:56,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:27:59,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:01,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:06,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:07,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:28:07,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:28:07,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:28:10,439 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:12,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:17,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:20,511 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:22,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:28:22,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:28:22,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:28:22,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:23,259 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:28:28,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:30,585 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:33,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:37,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:28:37,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:28:37,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:28:38,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:40,654 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:43,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:48,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:50,715 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:28:52,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:28:52,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:28:53,259 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:28:54,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:28:59,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:02,799 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:04,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:07,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:29:07,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:29:07,537 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:29:09,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:12,872 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:15,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:20,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:22,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:29:22,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:29:23,260 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:29:24,951 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:26,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:31,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:33,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:29:33,112 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:29:33,113 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:29:33,115 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:29:34,050 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:29:35,056 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:36,065 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:37,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:37,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:29:37,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:29:42,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:47,171 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:48,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:52,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:29:52,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:29:53,262 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:29:53,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:29:57,243 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:29:58,852 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:03,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:07,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:30:07,534 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:30:09,326 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:30:09,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:14,756 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:19,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:30:19,954 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:22,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:30:22,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:30:23,262 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:30:25,263 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:29,464 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:30:30,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:35,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:37,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:30:37,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:30:37,536 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:30:41,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:41,594 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:30:46,223 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:50,665 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:30:51,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:30:52,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:30:52,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:30:53,273 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:30:57,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:02,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:07,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:07,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:31:07,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:31:12,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:17,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:22,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:31:22,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:31:22,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:31:22,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:23,263 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:31:25,200 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:31:28,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:33,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:35,288 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:31:37,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:31:37,535 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:31:38,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:43,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:47,383 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:31:49,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:53,265 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:31:53,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:31:53,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:31:53,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:31:54,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:31:57,458 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:31:59,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:04,779 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:07,534 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:32:08,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:32:08,457 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:32:10,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:15,720 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:17,609 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:32:20,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:23,266 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:32:25,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:32:25,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:32:25,113 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:32:26,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:29,694 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:32:31,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:36,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:39,764 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:32:40,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:32:40,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:32:40,112 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:32:42,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:47,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:48,860 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:32:53,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:32:53,267 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:32:55,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:32:55,112 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:32:55,113 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:32:58,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:00,982 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:04,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:09,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:10,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:33:10,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:33:10,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:33:13,064 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:14,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:19,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:23,267 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:33:25,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:33:25,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:33:25,206 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:33:25,321 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:25,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:30,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:33,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:33:33,707 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:33:33,709 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:33:33,713 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:33:34,404 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:33:35,411 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:35,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:36,420 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:40,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:33:40,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:33:40,206 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:33:41,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:46,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:47,504 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:33:51,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:53,268 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:33:55,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:33:55,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:33:55,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:33:57,600 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:33:57,633 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:02,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:07,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:09,738 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:10,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:34:10,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:34:10,206 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:34:13,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:18,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:19,800 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:23,269 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:34:24,269 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:25,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:34:25,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:34:25,210 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:34:29,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:31,892 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:34,555 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:39,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:40,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:34:40,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:34:40,206 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:34:40,964 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:45,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:50,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:34:51,056 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:34:53,272 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:34:55,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:34:55,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:34:55,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:34:56,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:01,292 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:35:02,069 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:07,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:10,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:35:10,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:35:10,208 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:35:11,367 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:35:12,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:17,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:23,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:23,271 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:35:23,478 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:35:25,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:35:25,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:35:25,207 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:35:28,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:33,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:33,552 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:35:38,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:40,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:35:40,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:35:40,831 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:35:43,625 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:35:44,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:49,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:53,271 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:35:55,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:35:55,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:35:55,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:36:01,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:06,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:10,829 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:36:10,829 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:36:12,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:17,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:19,824 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:36:22,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:23,272 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:36:26,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:36:26,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:36:26,474 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:36:28,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:29,890 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:36:33,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:38,094 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:41,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:36:41,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:36:41,474 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:36:41,964 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:36:43,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:48,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:52,035 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:36:53,272 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:36:54,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:36:56,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:36:56,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:36:56,475 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:36:59,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:03,145 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:05,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:10,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:11,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:37:11,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:37:15,283 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:16,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:21,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:23,275 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:37:26,473 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:26,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:37:26,475 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:37:27,380 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:31,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:35,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:37:35,091 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:37:35,092 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:37:35,095 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:37:35,439 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:37:36,444 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:37,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:37,450 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:41,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:37:41,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:37:42,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:47,519 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:48,321 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:53,276 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:37:54,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:37:56,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:37:56,476 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:37:57,591 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:37:59,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:04,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:07,665 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:38:10,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:11,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:38:11,477 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:38:15,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:19,749 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:38:20,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:23,276 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:38:26,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:26,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:38:26,478 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:38:29,812 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:38:31,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:36,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:39,889 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:38:41,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:38:41,477 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:38:42,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:47,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:49,953 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:38:53,277 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:38:53,278 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:38:57,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:38:57,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:38:57,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:38:58,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:00,025 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:03,614 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:08,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:12,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:12,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:39:12,413 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:39:12,413 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:39:13,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:18,675 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:22,182 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:23,277 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:39:24,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:28,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:39:28,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:39:28,753 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:39:30,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:32,262 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:35,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:40,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:42,350 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:43,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:39:43,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:39:43,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:39:45,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:50,953 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:53,278 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:39:53,435 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:39:56,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:39:58,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:39:58,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:39:58,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:40:01,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:03,513 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:07,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:12,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:13,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:40:13,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:40:13,763 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:40:15,582 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:17,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:22,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:23,278 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:40:25,646 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:28,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:28,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:40:28,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:40:28,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:40:33,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:35,716 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:38,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:43,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:43,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:40:43,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:40:43,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:40:45,794 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:49,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:53,280 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:40:54,283 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:40:57,881 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:40:58,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:40:58,763 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:40:58,764 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:40:59,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:04,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:05,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:41:05,277 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:41:05,277 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:41:05,280 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:41:05,944 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:41:06,948 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:07,954 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:10,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:14,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:41:14,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:41:14,589 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:41:15,826 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:18,012 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:20,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:23,293 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:41:26,034 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:28,084 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:29,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:41:29,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:41:29,589 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:41:31,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:37,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:38,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:42,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:45,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:41:45,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:41:45,217 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:41:47,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:48,233 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:41:52,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:41:53,281 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:41:58,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:01,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:42:01,122 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:42:01,123 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:42:03,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:04,348 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:08,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:13,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:14,426 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:16,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:42:16,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:42:16,122 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:42:19,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:23,282 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:42:24,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:26,518 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:29,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:32,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:42:32,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:42:32,686 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:42:35,012 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:36,583 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:40,013 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:45,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:45,648 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:47,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:42:47,685 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:42:50,912 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:42:53,284 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:42:55,722 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:42:56,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:01,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:03,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:43:03,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:43:03,982 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:43:05,792 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:07,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:12,164 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:17,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:17,882 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:18,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:43:18,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:43:18,979 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:43:23,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:23,285 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:43:27,946 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:28,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:33,755 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:34,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:43:34,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:43:34,587 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:43:38,015 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:38,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:43,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:48,089 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:49,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:49,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:43:49,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:43:49,585 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:43:53,287 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:43:54,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:43:58,179 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:43:59,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:04,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:05,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:44:05,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:44:05,950 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:44:08,253 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:44:10,139 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:15,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:20,342 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:44:20,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:20,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:44:20,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:44:20,950 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:44:23,287 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:44:26,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:30,415 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:44:31,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:36,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:44:36,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:44:36,554 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:44:36,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:37,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:44:37,936 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:44:37,937 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:44:37,939 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:44:38,470 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:44:40,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:44:41,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:46,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:50,548 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:44:51,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:44:51,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:44:51,555 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:44:52,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:44:53,287 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:44:58,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:00,618 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:03,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:07,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:45:07,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:45:07,485 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:45:08,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:10,709 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:13,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:18,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:20,792 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:22,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:45:22,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:45:22,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:45:23,288 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:45:24,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:29,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:29,852 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:34,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:37,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:45:37,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:45:37,772 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:45:39,938 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:39,984 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:44,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:50,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:45:52,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:45:52,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:45:52,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:45:52,771 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:45:53,289 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:45:56,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:01,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:02,075 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:06,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:08,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:46:08,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:46:08,309 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:46:11,766 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:12,145 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:14,159 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:17,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:22,219 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:23,289 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:46:23,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:23,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:46:23,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:46:23,308 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:46:28,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:32,289 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:33,690 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:38,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:46:38,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:46:38,312 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:46:39,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:42,364 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:45,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:50,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:46:53,289 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:46:53,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:46:53,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:46:53,309 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:46:54,446 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:46:55,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:00,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:04,512 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:05,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:08,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:47:08,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:47:08,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:47:10,728 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:14,581 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:15,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:20,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:23,290 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:47:23,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:47:23,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:47:23,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:47:24,648 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:26,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:31,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:34,721 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:37,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:38,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:47:38,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:47:38,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:47:42,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:44,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:47,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:52,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:47:53,291 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:47:53,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:47:53,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:47:53,309 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:47:54,871 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:47:58,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:03,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:48:03,339 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:48:03,345 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:48:03,347 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:48:03,934 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:48:04,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:04,940 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:48:08,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:48:08,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:48:08,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:48:09,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:14,805 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:17,019 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:48:19,806 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:23,291 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:48:23,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:48:23,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:48:23,311 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:48:24,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:26,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:48:29,920 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:35,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:36,212 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:48:38,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:48:38,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:48:38,311 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:48:40,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:46,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:46,337 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:48:51,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:53,293 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:48:53,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:48:53,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:48:53,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:48:56,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:48:56,404 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:01,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:06,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:06,475 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:08,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:49:08,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:49:08,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:49:12,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:16,554 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:17,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:22,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:23,294 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:49:24,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:49:24,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:49:24,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:49:27,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:28,642 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:32,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:38,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:38,717 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:39,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:49:39,426 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:49:43,653 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:48,782 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:49:49,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:49:53,294 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:49:54,425 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:49:54,426 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:49:54,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:00,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:00,869 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:05,094 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:09,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:50:09,427 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:50:10,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:10,938 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:15,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:20,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:21,007 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:23,295 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:50:24,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:50:24,427 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:50:26,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:31,083 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:32,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:37,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:39,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:50:39,427 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:50:43,173 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:43,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:48,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:53,242 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:50:53,295 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:50:54,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:50:54,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:50:54,428 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:50:59,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:02,309 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:51:05,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:09,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:51:09,427 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:51:10,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:15,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:20,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:23,309 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:51:24,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:51:24,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:51:24,586 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:51:25,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:28,444 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:51:31,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:36,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:38,527 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:51:39,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:51:39,583 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:51:41,797 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:46,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:48,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:51:48,215 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:51:48,215 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:51:48,217 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:51:48,599 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:51:49,604 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:51:50,610 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:51:52,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:51:53,297 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:51:54,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:51:54,584 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:51:57,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:00,677 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:02,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:07,865 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:09,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:52:09,583 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:52:10,745 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:13,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:18,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:20,819 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:23,298 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:52:24,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:24,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:52:24,584 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:52:29,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:32,904 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:34,945 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:39,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:52:39,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:52:39,750 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:52:39,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:43,099 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:45,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:50,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:52:53,299 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:52:54,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:52:54,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:52:54,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:52:55,199 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:52:55,933 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:00,935 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:05,296 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:06,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:09,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:53:09,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:53:09,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:53:11,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:15,375 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:17,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:22,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:23,301 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:53:24,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:53:24,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:53:24,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:53:27,457 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:27,956 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:32,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:36,527 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:38,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:39,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:53:39,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:53:39,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:53:43,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:46,606 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:48,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:53,302 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:53:54,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:53:54,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:53:54,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:53:54,750 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:53:56,675 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:53:59,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:04,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:08,758 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:54:09,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:54:09,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:54:09,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:09,750 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:54:14,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:18,831 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:54:20,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:23,304 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:54:25,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:54:25,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:54:25,341 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:54:25,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:28,909 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:54:31,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:36,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:38,990 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:54:40,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:54:40,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:54:42,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:47,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:49,063 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:54:52,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:54:53,303 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:54:55,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:54:55,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:54:55,342 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:54:58,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:01,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:03,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:08,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:10,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:55:10,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:55:11,205 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:13,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:18,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:19,517 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:55:19,518 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:55:19,519 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:55:19,522 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:55:20,286 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:55:21,295 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:23,305 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:55:24,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:25,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:55:25,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:55:29,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:31,375 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:35,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:40,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:40,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:55:40,341 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:55:43,465 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:45,650 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:50,651 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:55:53,305 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:55:53,537 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:55:55,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:55:55,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:55:56,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:01,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:03,622 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:07,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:11,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:56:11,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:56:11,533 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:56:12,840 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:14,730 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:17,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:22,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:23,306 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:56:24,798 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:26,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:56:26,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:56:26,531 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:56:28,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:34,759 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:34,862 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:39,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:42,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:56:42,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:56:42,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:56:44,970 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:45,152 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:50,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:53,309 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:56:55,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:56:57,061 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:56:57,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:56:57,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:56:57,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:57:00,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:05,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:09,141 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:57:11,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:12,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:57:12,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:57:12,668 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:57:16,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:19,210 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:57:21,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:23,308 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:57:27,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:27,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:57:27,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:57:27,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:57:29,288 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:57:32,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:37,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:41,381 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:57:42,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:57:42,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:57:42,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:57:42,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:47,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:52,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:57:53,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:57:57,480 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:57:57,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:57:57,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:57:57,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:57:58,917 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:03,919 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:07,586 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:58:08,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:12,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:58:12,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:58:12,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:58:14,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:19,663 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:58:20,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:23,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:58:26,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:27,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:58:27,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:58:27,668 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:58:29,729 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:58:31,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:36,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:39,803 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:58:41,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:42,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:58:42,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:58:42,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:58:47,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:51,887 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:58:53,310 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:58:53,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:57,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:58:57,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:58:57,943 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:58:59,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:58:59,285 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 19:58:59,288 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 19:58:59,288 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 19:58:59,290 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 19:58:59,977 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 19:59:00,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:01,989 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:04,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:09,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:11,054 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:12,939 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:59:12,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:59:12,941 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:59:15,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:20,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:21,126 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:23,311 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:59:26,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:29,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 19:59:29,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:59:29,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:59:31,207 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:31,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:36,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:42,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:43,305 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:44,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:59:44,066 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 19:59:47,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:52,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:53,312 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 19:59:53,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 19:59:58,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 19:59:59,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 19:59:59,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:00:03,462 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:03,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:08,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:13,539 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:14,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:14,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:00:14,066 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:00:19,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:23,313 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:00:24,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:25,641 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:29,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:00:29,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:00:30,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:35,720 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:35,872 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:40,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:44,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:00:44,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:00:46,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:47,807 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:51,692 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:53,315 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:00:57,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:00:57,885 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:00:59,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:00:59,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:01:03,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:09,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:09,976 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:01:14,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:14,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:01:14,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:01:19,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:20,051 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:01:23,356 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:01:25,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:29,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:01:29,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:01:30,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:01:31,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:36,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:39,204 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:01:41,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:44,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:01:44,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:01:47,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:51,280 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:01:52,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:53,315 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:01:58,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:01:59,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:01:59,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:02:01,390 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:03,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:08,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:11,611 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:14,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:14,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:02:14,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:02:19,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:21,745 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:23,316 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:02:24,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:29,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:29,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:02:29,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:02:29,908 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:02:31,645 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:02:31,647 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:02:31,647 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:02:31,650 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:02:31,825 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:02:33,840 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:34,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:39,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:43,913 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:44,719 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:44,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:02:44,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:02:44,907 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:02:50,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:53,316 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:02:55,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:02:56,004 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:02:59,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:02:59,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:02:59,906 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:03:01,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:06,127 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:06,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:11,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:14,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:03:14,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:03:14,908 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:03:17,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:18,196 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:22,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:23,320 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:03:27,810 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:28,264 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:29,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:03:29,906 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:03:29,906 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:03:33,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:38,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:38,355 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:43,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:45,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:03:45,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:03:45,260 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:03:47,441 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:48,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:53,320 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:03:54,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:03:59,538 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:03:59,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:00,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:04:00,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:04:00,259 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:04:04,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:09,608 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:04:10,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:15,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:15,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:04:15,260 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:04:15,262 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:04:19,682 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:04:20,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:23,321 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:04:26,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:29,778 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:04:30,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:04:30,259 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:04:31,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:36,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:41,869 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:04:42,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:45,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:04:45,260 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:04:47,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:51,943 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:04:52,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:04:53,322 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:04:58,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:00,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:05:00,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:05:00,260 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:05:02,004 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:03,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:08,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:12,071 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:14,020 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:15,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:05:15,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:05:15,260 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:05:19,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:23,322 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:05:24,158 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:25,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:30,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:31,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:05:31,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:05:31,073 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:05:34,237 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:35,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:40,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:44,316 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:45,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:46,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:05:46,072 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:05:46,073 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:05:51,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:05:53,323 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:05:54,388 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:05:56,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:01,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:02,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:06:02,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:06:02,192 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:06:03,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:06:03,591 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:06:03,592 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:06:03,595 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:06:04,494 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:06:06,505 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:06:06,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:11,599 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:16,592 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:06:16,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:17,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:06:17,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:06:17,191 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:06:22,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:23,324 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:06:25,656 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:06:28,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:32,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:06:32,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:06:32,191 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:06:33,424 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:37,765 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:06:39,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:44,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:47,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:06:47,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:06:47,192 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:06:49,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:49,839 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:06:53,325 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:06:55,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:06:59,911 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:00,677 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:02,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:07:02,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:07:02,191 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:07:06,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:09,981 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:11,940 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:16,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:18,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:07:18,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:07:18,381 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:07:22,055 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:22,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:23,326 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:07:28,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:32,156 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:33,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:07:33,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:33,380 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:07:38,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:42,231 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:43,712 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:48,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:49,849 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:07:49,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:07:49,852 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:07:52,290 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:07:53,326 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:07:54,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:07:59,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:04,428 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:04,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:08:04,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:04,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:08:04,864 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:08:10,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:14,503 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:15,833 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:19,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:08:19,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:08:19,863 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:08:21,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:23,327 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:08:26,592 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:26,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:31,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:34,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:08:34,868 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:08:34,869 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:08:36,665 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:37,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:42,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:46,749 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:47,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:49,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:08:49,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:08:49,863 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:08:53,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:08:53,329 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:08:55,824 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:08:58,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:03,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:04,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:09:04,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:09:04,866 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:09:07,914 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:09:08,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:13,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:17,979 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:09:19,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:19,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:09:19,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:09:19,863 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:09:23,330 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:09:25,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:28,056 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:09:30,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:35,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:36,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:09:36,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:09:36,190 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:09:37,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:09:37,877 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:09:37,879 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:09:37,884 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:09:38,143 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:09:40,155 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:09:40,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:45,887 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:50,238 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:09:51,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:09:51,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:09:51,188 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:09:53,330 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:09:56,331 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:00,329 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:01,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:06,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:07,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:10:07,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:10:07,897 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:10:11,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:12,444 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:16,702 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:22,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:22,525 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:22,894 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:10:22,895 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:10:23,331 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:10:27,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:32,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:34,614 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:37,787 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:37,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:10:37,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:10:37,896 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:10:43,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:44,695 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:48,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:52,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:10:52,896 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:10:53,331 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:10:54,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:10:54,773 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:10:59,333 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:04,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:04,853 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:11:07,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:11:07,896 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:11:10,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:15,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:16,942 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:11:20,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:23,352 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:11:23,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:11:23,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:11:23,952 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:11:25,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:27,018 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:11:30,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:35,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:36,080 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:11:38,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:11:38,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:11:38,952 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:11:41,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:46,145 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:11:46,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:51,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:53,335 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:11:55,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:11:55,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:11:55,174 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:11:57,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:11:58,230 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:02,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:08,300 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:08,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:10,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:12:10,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:12:10,174 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:12:14,335 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:19,838 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:20,396 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:23,335 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:12:25,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:12:25,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:12:25,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:25,180 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:12:30,380 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:30,494 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:35,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:40,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:12:40,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:12:40,174 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:12:40,558 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:41,373 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:46,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:52,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:12:52,636 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:12:53,335 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:12:55,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:12:55,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:12:55,175 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:12:57,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:02,733 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:13:02,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:07,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:10,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:13:10,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:13:10,674 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:13:12,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:13:12,631 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:13:12,632 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:13:12,636 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:13:12,831 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:13:13,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:14,842 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:13:18,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:23,336 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:13:24,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:26,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:13:26,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:13:26,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:13:29,937 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:30,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:13:34,938 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:40,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:41,069 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:13:41,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:13:41,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:13:41,500 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:13:45,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:51,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:51,148 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:13:53,336 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:13:56,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:13:56,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:13:56,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:13:56,515 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:14:01,232 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:01,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:06,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:11,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:14:11,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:14:11,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:14:12,305 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:12,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:17,709 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:22,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:23,094 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:23,337 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:14:26,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:14:26,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:14:26,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:14:28,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:32,475 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:33,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:38,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:41,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:14:41,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:14:41,505 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:14:43,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:44,582 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:48,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:53,338 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:14:54,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:14:54,645 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:14:56,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:14:56,502 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:14:59,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:04,725 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:15:04,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:09,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:12,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:15:12,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:15:12,821 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:15:15,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:16,812 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:15:20,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:23,340 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:15:25,834 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:26,930 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:15:27,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:15:27,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:15:27,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:15:31,031 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:36,033 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:39,017 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:15:41,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:42,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:15:42,830 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:15:46,730 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:49,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:15:51,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:53,340 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:15:57,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:15:57,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:15:57,819 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:15:59,223 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:03,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:08,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:11,358 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:12,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:16:12,820 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:16:13,985 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:18,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:21,440 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:23,354 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:16:24,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:29,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:16:29,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:16:29,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:16:30,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:32,557 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:35,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:40,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:42,646 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:44,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:16:44,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:16:44,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:16:46,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:51,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:51,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:16:51,629 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:16:51,630 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:16:51,633 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:16:51,715 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:16:52,723 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:16:53,341 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:16:56,343 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:16:59,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:16:59,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:16:59,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:17:01,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:04,806 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:06,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:11,772 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:14,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:17:14,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:17:14,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:17:14,879 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:17,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:22,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:23,342 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:17:26,962 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:27,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:29,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:17:29,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:17:29,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:17:33,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:37,045 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:39,038 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:44,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:44,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:17:44,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:17:44,233 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:17:47,126 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:49,086 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:53,342 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:17:54,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:17:59,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:17:59,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:17:59,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:17:59,254 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:17:59,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:04,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:09,291 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:18:10,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:14,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:18:14,252 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:18:15,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:20,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:21,375 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:18:23,343 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:18:26,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:29,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:18:29,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:18:29,254 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:18:31,472 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:18:31,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:36,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:41,595 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:18:42,079 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:44,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:18:44,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:18:44,254 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:18:47,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:52,680 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:18:53,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:18:53,343 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:18:58,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:00,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:19:00,492 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:19:00,494 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:19:02,760 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:03,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:08,838 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:12,840 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:14,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:15,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:19:15,492 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:19:19,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:23,344 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:19:24,916 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:25,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:30,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:32,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:19:32,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:19:32,003 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:19:34,994 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:35,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:40,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:45,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:47,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:19:47,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:19:47,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:19:47,094 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:51,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:19:53,346 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:19:57,170 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:19:57,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:02,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:20:02,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:20:02,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:20:03,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:07,297 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:20:08,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:13,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:17,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:20:17,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:20:17,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:20:17,377 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:20:19,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:23,346 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:20:24,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:26,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:20:26,947 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:20:26,948 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:20:26,951 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:20:27,450 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:20:29,462 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:20:29,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:32,000 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:20:32,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:20:32,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:20:35,206 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:39,549 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:20:40,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:45,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:47,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:20:47,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:20:47,005 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:20:49,632 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:20:51,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:20:53,347 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:20:57,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:01,731 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:02,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:21:02,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:21:02,002 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:21:03,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:08,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:12,911 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:13,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:17,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:21:17,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:21:17,002 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:21:19,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:22,972 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:23,347 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:21:24,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:29,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:32,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:21:32,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:21:32,003 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:21:33,070 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:34,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:39,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:44,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:45,155 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:47,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:21:47,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:21:47,004 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:21:50,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:21:53,348 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:21:55,231 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:21:55,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:00,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:02,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:22:02,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:22:02,979 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:22:05,314 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:22:06,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:11,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:16,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:17,428 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:22:17,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:22:17,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:22:17,978 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:22:22,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:23,350 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:22:27,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:27,506 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:22:32,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:32,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:22:32,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:22:32,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:22:37,580 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:22:38,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:43,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:47,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:22:47,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:22:47,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:22:48,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:51,724 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:22:53,350 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:22:54,351 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:22:59,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:02,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:23:02,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:23:02,979 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:23:03,843 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:23:05,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:10,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:15,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:15,941 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:23:17,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:23:17,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:23:17,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:23:21,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:23,351 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:23:27,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:28,014 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:23:32,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:32,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:23:32,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:23:32,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:23:37,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:23:37,610 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:42,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:47,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:47,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:23:47,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:23:47,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:23:49,179 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:23:53,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:23:53,351 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:23:58,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:01,289 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:02,978 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:24:02,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:24:02,979 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:24:04,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:09,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:10,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:24:10,309 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:24:10,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:24:10,313 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:24:10,371 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:24:11,379 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:14,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:19,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:19,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:24:19,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:24:19,663 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:24:23,352 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:24:23,473 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:24,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:29,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:33,554 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:34,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:34,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:24:34,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:24:34,661 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:24:39,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:45,644 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:45,769 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:49,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:24:49,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:24:49,662 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:24:50,862 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:24:53,354 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:24:55,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:24:56,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:01,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:04,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:25:04,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:25:04,663 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:25:07,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:07,796 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:25:12,200 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:17,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:17,882 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:25:19,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:25:19,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:25:19,662 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:25:22,911 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:23,356 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:25:27,948 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:25:28,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:33,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:34,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:25:34,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:25:34,664 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:25:38,028 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:25:38,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:43,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:48,090 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:25:49,633 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:25:49,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:25:49,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:25:49,662 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:25:53,356 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:25:55,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:00,175 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:01,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:04,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:26:04,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:26:04,662 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:26:06,847 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:10,257 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:12,660 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:17,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:19,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:26:19,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:26:19,663 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:26:21,330 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:22,893 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:23,371 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:26:28,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:31,406 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:33,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:34,661 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:26:34,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:26:34,663 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:26:38,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:41,532 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:43,933 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:48,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:50,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:26:50,341 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:26:50,342 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:26:53,359 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:26:53,646 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:26:54,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:26:59,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:03,718 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:04,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:05,338 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:27:05,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:27:09,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:15,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:15,800 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:20,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:20,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:27:20,342 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:27:23,360 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:27:25,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:25,864 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:30,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:35,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:27:35,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:27:35,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:36,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:41,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:45,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:27:45,185 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:27:45,186 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:27:45,189 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:27:46,012 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:46,012 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:27:47,192 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:48,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:27:50,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:27:50,339 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:27:52,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:53,361 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:27:57,833 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:27:58,098 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:02,834 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:05,872 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:28:05,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:28:05,874 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:28:08,217 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:08,261 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:13,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:19,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:20,351 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:20,873 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:28:20,875 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:28:23,362 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:28:24,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:30,202 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:30,416 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:35,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:37,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:28:37,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:28:37,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:28:39,485 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:40,724 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:45,725 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:50,897 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:28:51,595 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:28:52,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:28:52,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:28:52,348 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:28:53,362 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:28:56,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:01,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:01,693 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:29:06,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:07,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:29:07,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:29:07,350 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:29:11,777 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:29:11,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:16,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:21,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:22,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:29:22,350 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:29:23,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:29:27,587 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:27,884 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:29:32,590 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:37,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:29:37,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:29:37,971 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:29:38,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:43,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:48,051 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:29:48,864 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:52,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:29:52,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:29:53,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:29:54,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:29:59,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:00,134 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:04,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:07,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:30:07,350 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:30:09,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:10,202 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:14,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:20,273 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:20,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:22,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:30:22,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:30:23,364 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:30:26,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:31,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:32,365 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:36,835 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:37,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:30:37,349 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:30:42,438 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:42,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:47,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:52,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:30:52,361 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:30:52,362 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:30:52,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:30:53,365 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:30:54,517 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:30:58,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:03,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:04,589 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:31:07,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:31:07,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:31:08,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:13,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:16,672 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:31:18,593 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:22,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:31:22,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:31:22,566 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:31:23,366 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:31:23,835 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:23,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:31:23,877 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:31:23,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:31:23,884 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:31:24,759 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:31:25,764 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:31:28,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:33,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:35,836 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:31:37,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:31:37,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:31:37,606 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:31:39,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:44,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:45,912 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:31:50,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:52,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:31:52,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:31:52,564 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:31:53,368 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:31:55,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:31:58,007 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:01,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:06,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:07,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:32:07,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:32:07,561 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:32:08,079 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:11,801 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:16,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:18,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:22,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:22,560 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:32:22,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:32:22,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:32:23,368 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:32:27,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:30,239 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:32,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:37,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:32:37,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:32:37,564 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:32:38,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:40,326 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:43,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:48,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:50,405 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:32:52,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:32:52,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:32:52,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:32:53,368 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:32:54,370 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:32:59,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:02,484 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:04,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:07,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:33:07,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:33:07,562 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:33:09,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:14,565 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:14,823 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:19,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:22,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:33:22,566 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:33:22,567 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:33:23,370 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:33:24,647 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:25,371 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:30,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:34,728 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:36,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:37,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:33:37,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:33:37,568 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:33:41,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:46,801 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:47,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:52,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:33:52,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:33:52,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:33:52,953 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:33:53,371 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:33:56,883 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:33:58,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:03,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:05,947 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:34:07,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:34:07,951 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:34:09,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:14,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:16,023 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:34:19,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:22,952 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:34:22,953 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:34:22,954 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:34:23,373 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:34:25,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:28,109 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:34:31,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:36,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:37,952 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:34:37,953 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:34:38,181 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:34:42,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:47,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:50,264 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:34:52,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:52,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:34:52,957 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:34:53,374 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:34:58,205 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:34:58,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:34:58,246 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:34:58,247 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:34:58,248 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:34:58,326 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:35:00,336 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:03,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:07,953 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:35:07,955 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:35:09,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:10,445 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:14,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:19,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:22,562 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:22,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:35:22,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:35:22,959 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:35:23,374 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:35:25,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:30,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:32,655 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:35,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:37,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:35:37,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:35:37,962 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:35:41,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:42,825 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:46,055 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:51,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:35:52,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:35:52,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:35:52,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:35:52,961 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:35:53,375 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:35:56,376 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:01,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:04,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:07,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:07,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:36:07,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:36:07,960 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:36:12,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:17,089 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:17,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:22,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:23,377 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:36:23,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:36:23,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:36:23,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:36:26,146 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:27,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:32,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:36,225 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:38,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:38,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:36:38,391 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:36:43,605 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:46,372 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:48,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:53,378 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:36:53,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:36:53,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:36:53,395 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:36:54,607 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:36:58,458 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:36:59,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:04,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:08,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:37:08,392 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:37:08,523 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:37:10,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:15,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:18,604 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:37:21,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:23,378 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:37:23,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:37:23,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:37:23,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:37:26,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:28,686 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:37:32,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:37,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:38,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:37:38,392 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:37:40,791 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:37:42,617 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:47,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:50,882 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:37:52,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:37:53,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:37:53,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:37:53,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:37:58,601 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:00,950 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:03,716 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:08,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:09,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:38:09,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:38:09,032 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:38:13,044 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:14,382 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:19,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:23,106 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:23,379 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:38:24,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:38:24,029 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:38:24,029 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:38:25,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:30,233 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:31,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:38:31,366 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:38:31,367 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:38:31,370 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:38:32,178 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:32,178 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:38:33,183 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:35,372 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:40,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:38:40,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:38:40,370 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:38:40,608 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:43,259 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:46,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:51,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:38:53,327 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:38:53,380 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:38:55,368 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:38:55,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:38:55,369 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:38:56,586 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:01,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:04,399 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:39:07,203 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:11,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:39:11,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:39:11,880 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:39:13,119 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:14,461 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:39:18,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:23,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:23,381 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:39:24,542 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:39:26,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:39:26,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:39:26,880 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:39:29,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:34,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:34,628 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:39:39,191 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:41,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:39:41,883 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:39:41,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:39:45,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:46,726 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:39:50,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:53,382 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:39:56,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:39:56,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:39:56,880 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:39:58,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:40:02,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:07,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:08,921 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:40:11,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:40:11,881 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:40:13,047 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:18,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:23,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:23,382 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:40:26,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:40:26,881 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:40:28,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:33,077 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:37,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:40:38,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:41,880 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:40:41,881 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:40:44,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:47,462 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:40:49,369 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:53,383 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:40:54,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:40:58,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:40:58,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:40:58,136 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:40:59,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:00,565 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:41:04,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:09,657 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:12,662 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:41:13,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:41:13,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:41:13,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:41:15,381 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:20,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:22,735 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:41:23,384 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:41:25,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:28,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:41:28,135 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:41:31,308 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:34,835 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:41:36,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:41,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:43,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:41:43,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:41:43,699 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:41:46,917 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:41:47,592 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:52,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:53,403 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:41:58,186 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:41:58,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:41:58,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:41:58,698 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:41:59,008 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:03,879 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:09,079 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:09,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:13,697 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:42:13,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:42:13,699 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:42:14,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:20,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:21,163 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:23,386 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:42:26,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:28,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:42:28,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:42:28,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:42:30,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:42:30,522 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:42:30,523 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:42:30,526 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:42:31,236 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:42:31,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:32,246 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:33,251 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:36,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:41,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:43,325 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:43,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:42:43,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:42:47,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:52,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:53,387 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:42:55,415 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:42:57,773 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:42:58,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:42:58,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:43:03,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:05,498 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:43:08,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:13,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:13,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:43:13,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:43:13,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:43:17,589 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:43:19,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:23,389 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:43:24,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:27,664 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:43:28,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:43:28,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:43:30,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:35,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:38,762 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:43:40,761 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:43,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:43:43,980 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:43:46,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:48,831 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:43:51,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:53,389 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:43:56,390 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:43:58,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:43:58,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:43:58,985 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:44:00,914 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:02,250 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:07,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:10,976 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:12,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:13,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:44:13,981 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:44:18,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:21,052 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:23,269 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:23,390 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:44:28,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:29,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:44:29,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:44:29,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:44:33,134 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:33,912 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:38,913 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:43,222 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:44,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:44:44,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:44:44,499 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:44:44,658 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:49,659 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:53,294 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:44:53,390 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:44:55,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:44:59,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:44:59,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:44:59,503 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:45:00,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:05,395 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:45:05,818 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:10,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:14,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:45:14,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:45:14,500 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:45:15,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:45:16,680 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:21,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:23,392 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:45:27,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:27,558 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:45:29,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:45:29,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:45:29,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:45:32,714 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:37,632 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:45:38,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:43,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:44,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:45:44,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:45:44,501 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:45:48,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:53,392 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:45:53,743 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:45:54,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:45:59,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:00,597 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:46:00,615 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:46:00,624 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:46:03,823 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:04,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:09,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:12,870 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:46:12,872 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:46:12,872 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:46:12,875 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:46:12,899 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:46:14,912 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:15,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:46:15,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:15,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:46:15,665 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:46:15,915 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:20,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:23,393 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:46:24,977 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:26,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:30,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:46:30,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:46:30,665 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:46:31,861 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:36,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:37,176 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:41,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:45,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:46:45,676 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:46:47,251 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:47,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:52,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:46:53,397 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:46:57,318 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:46:58,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:00,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:47:00,674 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:47:03,920 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:07,470 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:47:09,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:14,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:15,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:47:15,676 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:47:19,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:19,567 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:47:23,394 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:47:24,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:29,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:29,655 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:47:30,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:47:30,675 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:47:34,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:39,760 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:47:39,915 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:44,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:45,674 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:47:45,675 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:47:50,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:47:51,839 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:47:53,395 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:47:56,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:00,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:48:00,687 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:48:01,906 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:48:01,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:06,945 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:11,974 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:48:12,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:15,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:48:15,687 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:48:17,933 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:23,290 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:23,396 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:48:24,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:48:28,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:30,686 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:48:30,687 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:48:33,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:35,185 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:48:39,624 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:44,626 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:45,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:48:45,687 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:48:47,271 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:48:49,924 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:53,398 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:48:55,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:48:57,346 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:00,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:00,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:49:00,688 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:49:05,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:07,488 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:10,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:15,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:15,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:49:15,688 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:49:17,551 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:21,078 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:23,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:49:26,402 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:27,640 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:30,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:49:30,688 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:49:31,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:36,888 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:39,725 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:42,328 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:46,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:49:46,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:49:46,062 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:49:47,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:47,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:49:47,435 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:49:47,436 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:49:47,439 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:49:47,786 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:49:48,790 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:49,796 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:49:52,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:53,399 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:49:57,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:49:59,864 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:01,060 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:50:01,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:50:01,061 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:50:03,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:09,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:09,952 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:14,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:17,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:50:17,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:50:17,042 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:50:19,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:20,021 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:23,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:50:25,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:30,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:32,093 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:32,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:50:32,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:50:32,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:50:36,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:41,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:42,167 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:46,890 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:47,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:50:47,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:50:47,103 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:50:52,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:50:52,239 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:50:53,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:50:57,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:02,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:51:02,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:51:03,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:04,328 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:51:08,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:13,393 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:51:14,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:17,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:51:17,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:51:20,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:23,400 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:51:23,458 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:51:25,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:30,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:32,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:51:32,102 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:51:32,103 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:51:35,535 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:51:36,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:41,363 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:46,364 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:47,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:51:47,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:51:52,277 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:51:53,403 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:51:55,754 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:51:57,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:02,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:52:02,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:52:02,354 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:07,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:07,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:52:12,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:17,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:52:17,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:52:17,934 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:52:18,295 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:23,296 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:23,403 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:52:28,006 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:52:28,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:32,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:52:32,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:52:34,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:38,072 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:52:39,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:44,708 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:47,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:52:47,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:52:48,152 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:52:50,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:53,404 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:52:55,405 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:52:58,237 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:01,209 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:02,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:53:02,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:53:06,261 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:10,314 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:11,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:16,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:17,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:53:17,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:53:17,233 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:53:20,389 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:22,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:23,405 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:53:28,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:29,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:53:29,300 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:53:29,301 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:53:29,305 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:53:29,443 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:53:30,450 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:31,454 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:32,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:53:32,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:53:32,229 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:53:33,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:38,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:42,531 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:43,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:47,231 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:53:47,232 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:53:47,234 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:53:49,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:53:51,608 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:53:53,406 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:53:55,407 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:00,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:01,728 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:02,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:54:02,232 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:54:05,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:10,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:11,799 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:16,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:17,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:54:17,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:54:21,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:21,871 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:23,406 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:54:27,408 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:32,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:54:32,231 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:54:32,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:33,964 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:37,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:43,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:44,041 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:47,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:54:47,230 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:54:48,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:53,408 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:54:53,802 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:54:56,127 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:54:58,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:03,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:55:03,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:55:03,295 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:55:04,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:06,201 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:09,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:14,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:16,272 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:18,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:55:18,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:55:18,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:55:20,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:23,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:55:26,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:28,372 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:31,075 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:33,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:55:33,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:55:33,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:55:36,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:38,443 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:41,790 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:46,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:48,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:55:48,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:55:48,509 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:52,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:55:53,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:55:55,546 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:56,550 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:55:58,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:01,579 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:02,585 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:03,292 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:03,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:56:03,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:56:08,453 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:12,660 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:13,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:18,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:56:18,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:56:18,295 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:56:19,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:22,739 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:23,409 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:56:25,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:30,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:31,893 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:33,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:56:33,294 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:56:35,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:40,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:44,049 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:45,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:48,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:56:48,294 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:56:51,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:56:52,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 20:56:52,956 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 20:56:52,957 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 20:56:52,960 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 20:56:53,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:53,108 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 20:56:53,410 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:56:54,112 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:56:57,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:02,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:03,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:57:03,294 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:57:04,184 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:57:07,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:12,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:14,259 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:57:17,788 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:18,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:57:18,294 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:57:23,411 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:57:23,412 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:26,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:57:29,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:33,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:57:33,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:57:33,342 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:57:34,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:36,443 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:57:39,627 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:44,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:46,519 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:57:48,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:57:48,343 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:57:48,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:57:50,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:53,413 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:57:56,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:57:58,593 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:01,561 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:03,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:58:03,340 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:58:06,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:08,685 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:11,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:16,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:18,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:58:18,341 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:58:18,778 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:22,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:23,413 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:58:28,310 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:30,859 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:33,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:33,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:58:33,341 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:58:38,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:40,932 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:43,988 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:48,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:58:48,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:58:48,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:58:49,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:58:50,004 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:58:53,413 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:58:55,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:01,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:02,094 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:03,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:59:03,432 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:59:06,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:12,172 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:12,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:17,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:18,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:59:18,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:59:18,447 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:59:22,235 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:22,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:23,414 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:59:28,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:33,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:33,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:59:33,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:59:34,323 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:38,603 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:43,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:44,398 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:48,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 20:59:48,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 20:59:48,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 20:59:49,698 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:53,414 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 20:59:54,469 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 20:59:54,823 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 20:59:59,824 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:03,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:00:03,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:00:03,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:00:04,533 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:04,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:09,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:15,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:16,625 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:18,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:00:18,435 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:00:20,637 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:23,417 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:00:24,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:00:24,718 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:00:24,719 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:00:24,723 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:00:25,703 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:25,703 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:00:25,726 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:26,710 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:30,727 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:34,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:00:34,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:00:34,156 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:00:36,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:36,794 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:41,287 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:47,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:48,879 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:00:49,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:00:49,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:00:49,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:00:52,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:53,417 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:00:58,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:00:58,950 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:01:03,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:04,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:01:04,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:01:04,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:01:08,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:09,018 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:01:13,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:18,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:19,169 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:01:19,170 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:01:19,171 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:01:21,098 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:01:23,414 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:23,417 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:01:28,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:33,176 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:01:34,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:01:34,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:01:34,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:39,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:44,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:46,273 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:01:49,168 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:01:49,169 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:01:50,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:53,419 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:01:55,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:01:56,349 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:00,718 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:05,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:02:05,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:02:05,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:02:06,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:08,426 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:11,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:16,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:18,500 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:20,065 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:02:20,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:02:20,066 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:02:22,234 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:23,419 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:02:27,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:28,580 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:32,780 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:35,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:02:35,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:02:35,069 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:02:38,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:40,667 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:43,159 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:48,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:50,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:02:50,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:02:50,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:02:50,738 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:02:53,329 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:02:53,420 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:02:58,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:00,806 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:03,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:05,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:03:05,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:03:05,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:03:09,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:10,879 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:14,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:19,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:20,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:03:20,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:03:20,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:03:20,963 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:23,421 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:03:24,421 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:29,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:33,092 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:35,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:03:35,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:03:35,067 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:03:35,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:40,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:43,162 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:45,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:50,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:03:50,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:03:50,071 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:03:51,349 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:03:53,248 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:03:53,421 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:03:56,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:01,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:01,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:04:01,920 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:04:01,921 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:04:01,925 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:04:02,313 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:04:03,319 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:04,323 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:05,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:04:05,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:04:05,068 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:04:07,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:13,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:15,408 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:18,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:21,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:04:21,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:04:21,268 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:04:23,421 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:04:23,422 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:24,605 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:28,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:34,327 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:34,680 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:36,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:04:36,267 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:04:39,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:44,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:44,757 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:04:49,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:52,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:04:52,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:04:52,761 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:04:53,423 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:04:55,322 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:04:56,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:00,323 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:06,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:06,925 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:07,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:05:07,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:05:07,760 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:05:11,945 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:16,997 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:17,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:22,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:23,426 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:05:24,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:05:24,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:05:24,290 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:05:27,062 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:27,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:32,671 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:38,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:39,153 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:39,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:05:39,289 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:05:43,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:48,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:49,260 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:05:53,425 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:05:54,426 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:05:55,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:05:55,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:05:55,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:05:59,328 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:00,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:01,340 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:05,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:10,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:10,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:06:10,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:06:10,884 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:06:13,420 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:16,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:21,388 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:23,426 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:06:23,496 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:25,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:06:25,882 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:06:25,882 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:06:27,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:32,123 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:35,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:37,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:41,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:06:41,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:06:41,247 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:06:42,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:44,660 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:48,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:53,160 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:06:53,426 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:06:54,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:06:56,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:06:56,246 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:06:58,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:03,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:06,847 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:08,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:11,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:07:11,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:07:11,247 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:07:14,478 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:16,919 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:20,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:23,428 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:07:25,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:26,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:07:26,248 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:07:26,987 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:30,496 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:35,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:36,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:07:36,246 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:07:36,246 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:07:36,249 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:07:37,067 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:37,067 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:07:38,072 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:41,246 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:41,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:07:41,247 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:07:46,511 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:49,149 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:07:51,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:53,428 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:07:56,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:07:56,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:07:56,249 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:07:57,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:07:59,221 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:02,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:07,158 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:09,298 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:11,247 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:08:11,248 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:08:12,490 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:17,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:19,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:23,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:23,429 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:08:26,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:08:26,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:08:26,815 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:08:28,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:29,440 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:33,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:38,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:41,523 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:41,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:08:41,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:08:41,813 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:08:43,987 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:49,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:51,597 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:08:53,430 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:08:55,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:08:57,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:08:57,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:08:57,901 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:09:01,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:01,679 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:06,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:11,763 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:12,093 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:12,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:09:12,898 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:09:17,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:22,774 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:23,431 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:09:23,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:27,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:09:27,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:09:27,901 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:09:28,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:33,051 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:33,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:38,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:42,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:09:42,899 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:09:44,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:45,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:49,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:53,430 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:09:54,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:09:55,235 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:09:57,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:09:57,899 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:10:00,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:05,297 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:10:05,953 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:10,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:13,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:10:13,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:10:13,003 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:10:15,366 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:10:16,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:21,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:23,431 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:10:27,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:27,534 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:10:28,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:10:28,001 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:10:28,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:10:32,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:37,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:37,609 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:10:42,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:43,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:10:43,004 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:10:43,006 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:10:47,680 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:10:48,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:53,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:53,433 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:10:58,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:10:58,002 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:10:58,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:10:59,773 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:03,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:07,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:11:07,147 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:11:07,148 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:11:07,152 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:11:07,844 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:11:08,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:09,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:09,854 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:13,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:11:13,004 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:11:14,182 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:19,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:19,929 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:23,446 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:11:24,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:28,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:11:28,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:11:28,004 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:11:30,009 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:30,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:35,305 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:39,074 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:40,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:43,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:11:43,003 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:11:46,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:51,171 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:11:51,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:53,434 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:11:57,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:11:58,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:11:58,004 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:12:01,243 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:02,820 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:07,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:11,322 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:13,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:12:13,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:13,004 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:12:18,202 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:21,395 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:23,435 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:12:23,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:28,437 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:29,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:12:29,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:12:29,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:12:33,484 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:33,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:38,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:43,556 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:44,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:12:44,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:12:44,092 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:12:44,273 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:49,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:53,435 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:12:53,651 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:12:54,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:12:59,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:12:59,114 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:12:59,134 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:13:00,410 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:03,734 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:06,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:11,251 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:14,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:13:14,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:13:14,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:13:15,814 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:16,332 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:21,334 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:23,436 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:13:25,892 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:26,844 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:29,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:13:29,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:13:29,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:13:32,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:35,979 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:38,025 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:43,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:44,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:13:44,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:13:44,092 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:13:46,050 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:48,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:53,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:53,438 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:13:56,119 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:13:58,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:13:59,090 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:13:59,091 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:13:59,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:14:04,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:08,241 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:10,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:14,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:14:14,544 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:14:14,545 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:14:15,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:17,313 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:21,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:23,440 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:14:26,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:27,388 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:29,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:14:29,542 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:14:31,750 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:36,752 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:37,086 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:14:37,088 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:14:37,088 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:14:37,091 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:14:37,457 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:14:38,462 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:39,468 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:42,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:46,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:14:46,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:14:46,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:14:47,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:49,551 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:14:52,777 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:53,439 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:14:58,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:14:59,656 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:01,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:15:01,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:15:01,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:15:04,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:10,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:11,749 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:15,105 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:16,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:15:16,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:15:16,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:15:20,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:21,816 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:23,440 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:15:25,441 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:31,050 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:31,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:15:31,166 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:15:31,894 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:36,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:41,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:41,957 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:46,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:15:46,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:15:46,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:15:47,347 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:52,032 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:15:52,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:15:53,440 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:15:58,442 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:01,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:16:01,167 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:16:01,168 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:16:03,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:04,101 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:08,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:14,169 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:14,434 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:16,166 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:16:16,167 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:16:20,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:23,453 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:16:24,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:25,454 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:30,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:31,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:16:31,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:16:31,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:16:34,316 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:35,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:40,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:46,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:46,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:16:46,494 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:16:46,501 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:51,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:16:53,442 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:16:55,576 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:16:57,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:02,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:02,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:17:02,875 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:17:05,662 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:17:08,395 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:13,396 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:17,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:17:17,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:17:17,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:17:19,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:21,781 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:17:23,442 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:17:24,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:29,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:31,861 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:17:32,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:17:32,875 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:17:35,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:40,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:41,935 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:17:45,209 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:47,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:17:47,875 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:17:51,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:17:53,443 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:17:54,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:17:56,444 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:01,446 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:02,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:18:02,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:18:04,089 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:07,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:12,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:12,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:18:12,271 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:18:12,272 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:18:12,274 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:18:12,329 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:18:13,335 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:14,340 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:17,276 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:17,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:18:17,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:18:23,007 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:23,444 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:18:24,407 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:28,445 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:32,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:18:32,877 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:18:32,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:18:33,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:36,494 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:38,808 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:43,810 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:46,562 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:47,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:18:47,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:18:49,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:53,446 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:18:54,448 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:18:56,625 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:18:59,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:03,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:19:03,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:19:03,901 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:19:05,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:06,686 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:19:10,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:15,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:17,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:19:18,898 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:19:18,899 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:19:18,899 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:19:21,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:23,448 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:19:26,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:27,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:19:31,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:34,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:19:34,706 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:19:34,708 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:19:37,061 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:37,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:19:42,062 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:47,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:48,011 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:19:49,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:19:49,705 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:19:52,892 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:53,447 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:19:57,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:19:58,080 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:02,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:05,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:20:05,761 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:20:08,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:08,212 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:13,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:18,602 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:20,300 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:20,760 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:20:20,760 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:20:23,448 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:20:24,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:29,770 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:30,370 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:34,771 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:36,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:20:36,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:20:36,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:20:40,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:40,548 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:45,106 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:50,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:20:50,857 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:20:51,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:20:51,669 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:20:53,448 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:20:56,449 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:01,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:06,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:06,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:21:06,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:21:06,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:21:11,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:12,796 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:16,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:21,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:21:21,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:21:21,716 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:21:21,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:23,449 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:21:24,878 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:27,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:32,786 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:33,947 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:36,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:21:36,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:21:36,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:21:37,909 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:43,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:44,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:48,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:51,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:21:51,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:21:51,862 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:21:53,423 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:21:53,424 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:21:53,425 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:21:53,443 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:21:53,449 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:21:54,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:54,140 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:21:54,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:21:55,145 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:21:59,451 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:04,681 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:06,222 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:06,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:22:06,860 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:22:10,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:15,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:16,295 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:20,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:23,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:22:23,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:22:23,072 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:22:23,451 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:22:26,380 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:26,835 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:31,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:36,452 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:36,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:38,057 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:22:38,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:22:38,059 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:22:42,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:46,532 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:48,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:53,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:22:53,453 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:22:54,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:22:54,307 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:22:54,311 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:22:58,674 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:22:58,943 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:03,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:08,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:23:09,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:23:09,307 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:23:09,489 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:14,491 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:18,845 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:23:20,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:23,453 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:23:24,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:23:24,306 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:23:25,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:30,937 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:23:31,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:36,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:39,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:23:39,307 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:23:41,008 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:23:41,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:46,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:51,791 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:23:53,081 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:23:53,454 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:23:54,306 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:23:54,307 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:23:57,497 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:02,150 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:02,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:07,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:10,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:24:10,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:24:10,380 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:24:12,214 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:13,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:18,673 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:23,454 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:24:24,291 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:24,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:25,377 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:24:25,378 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:24:25,378 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:24:29,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:34,375 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:34,606 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:39,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:41,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:24:41,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:24:41,481 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:24:44,448 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:44,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:49,870 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:53,455 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:24:54,522 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:24:55,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:24:56,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:24:56,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:24:56,482 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:25:00,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:05,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:06,608 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:10,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:12,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:25:12,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:25:12,674 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:25:16,019 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:16,692 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:21,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:23,455 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:25:24,157 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:25:24,158 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:25:24,159 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:25:24,162 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:25:24,761 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:25:25,766 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:26,165 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:26,772 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:27,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:25:27,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:25:27,669 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:25:31,871 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:36,847 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:37,248 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:42,249 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:43,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:25:43,440 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:25:43,443 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:25:46,917 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:47,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:52,795 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:25:53,458 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:25:56,997 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:25:58,438 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:25:58,439 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:25:58,439 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:25:58,619 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:03,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:07,063 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:26:08,689 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:13,691 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:14,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:26:14,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:26:14,656 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:26:19,045 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:19,163 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:26:23,472 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:26:24,474 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:29,237 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:26:29,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:26:29,652 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:26:29,653 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:26:29,807 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:34,809 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:39,311 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:26:40,471 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:45,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:45,812 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:26:45,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:26:45,815 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:26:48,386 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:26:51,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:53,459 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:26:56,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:26:58,456 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:00,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:27:00,814 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:27:00,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:27:02,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:07,054 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:08,519 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:12,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:15,814 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:27:15,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:27:18,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:20,602 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:23,303 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:23,459 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:27:28,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:30,667 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:30,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:27:30,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:27:30,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:27:33,979 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:38,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:40,745 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:44,891 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:45,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:27:45,816 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:27:50,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:27:52,822 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:27:53,460 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:27:55,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:00,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:00,817 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:28:00,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:28:02,889 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:05,643 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:10,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:14,972 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:15,815 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:28:15,816 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:15,817 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:28:21,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:23,461 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:28:25,042 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:26,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:31,465 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:31,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:28:31,559 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:28:31,561 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:28:35,107 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:37,030 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:42,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:45,189 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:46,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:28:46,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:28:46,572 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:28:47,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:52,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:28:53,462 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:28:54,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:28:54,236 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:28:54,237 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:28:54,239 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:28:54,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:28:55,259 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:57,268 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:28:58,241 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:03,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:29:03,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:29:03,103 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:29:03,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:07,354 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:29:08,642 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:13,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:17,423 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:29:18,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:29:18,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:29:19,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:23,465 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:29:24,467 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:27,498 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:29:30,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:34,429 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:29:34,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:29:34,431 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:29:35,644 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:36,660 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:29:40,823 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:45,825 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:48,751 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:29:49,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:29:49,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:29:49,432 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:29:51,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:53,466 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:29:56,994 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:29:58,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:01,995 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:04,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:30:04,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:30:04,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:30:07,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:08,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:12,688 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:17,690 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:18,983 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:19,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:30:19,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:30:19,431 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:30:22,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:23,466 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:30:28,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:29,069 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:33,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:34,430 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:30:34,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:30:38,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:41,241 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:43,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:48,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:49,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:30:49,432 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:30:51,334 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:30:53,467 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:30:54,468 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:30:59,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:03,414 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:04,431 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:31:04,431 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:31:04,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:09,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:13,485 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:15,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:20,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:20,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:31:20,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:31:20,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:31:23,479 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:31:25,567 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:26,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:31,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:34,733 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:35,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:31:35,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:31:35,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:31:37,173 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:42,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:44,810 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:47,717 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:52,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:31:52,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:31:52,214 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:31:53,468 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:31:53,469 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:31:54,879 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:31:58,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:03,596 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:04,962 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:07,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:32:07,212 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:32:09,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:14,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:17,049 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:19,571 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:23,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:32:23,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:32:23,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:32:23,646 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:32:24,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:24,848 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:32:24,849 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:32:24,850 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:32:24,853 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:32:25,110 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:32:26,126 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:27,133 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:29,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:34,856 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:37,223 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:38,459 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:32:38,460 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:32:40,683 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:45,684 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:51,328 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:32:51,401 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:32:53,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:32:53,459 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:32:53,628 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:32:56,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:01,793 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:03,412 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:06,794 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:09,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:33:09,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:33:09,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:33:11,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:13,481 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:16,889 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:22,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:23,470 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:33:23,617 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:24,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:33:24,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:33:27,762 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:33,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:35,709 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:38,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:39,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:33:39,538 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:33:43,792 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:45,786 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:49,063 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:53,473 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:33:54,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:33:54,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:33:54,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:33:54,541 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:33:57,894 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:33:59,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:04,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:09,000 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:34:09,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:34:09,539 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:34:10,703 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:15,705 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:21,081 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:34:21,304 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:23,473 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:34:24,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:34:24,540 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:34:26,733 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:31,152 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:34:32,175 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:37,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:39,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:34:39,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:34:39,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:34:42,324 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:43,238 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:34:47,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:52,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:34:53,473 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:34:54,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:34:54,665 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:34:55,352 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:34:57,827 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:02,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:05,431 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:35:08,375 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:09,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:35:09,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:35:09,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:35:13,834 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:17,528 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:35:19,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:23,474 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:35:24,477 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:24,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:35:24,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:35:24,672 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:35:27,610 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:35:30,171 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:35,172 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:37,694 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:35:39,665 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:35:39,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:35:40,881 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:45,883 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:49,846 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:35:51,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:53,474 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:35:54,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:35:54,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:35:54,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:35:56,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:35:59,938 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:02,080 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:07,081 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:09,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:36:09,201 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:36:09,201 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:36:09,204 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:36:09,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:36:09,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:36:10,012 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:10,013 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:36:12,021 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:12,822 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:17,823 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:22,084 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:23,475 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:36:23,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:24,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:36:24,666 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:36:28,863 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:31,164 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:34,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:39,180 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:39,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:36:39,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:36:39,668 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:36:41,248 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:44,628 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:49,630 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:53,322 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:36:53,477 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:36:54,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:36:54,667 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:36:54,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:36:59,869 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:03,399 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:05,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:10,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:11,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:37:11,259 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:37:11,261 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:37:13,476 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:15,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:20,546 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:23,477 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:37:23,542 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:26,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:37:26,257 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:26,258 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:37:31,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:35,618 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:37,002 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:41,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:37:41,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:37:41,857 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:37:42,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:45,684 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:47,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:52,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:37:53,477 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:37:55,762 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:37:56,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:37:56,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:37:58,064 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:03,066 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:05,863 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:38:08,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:11,853 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:38:11,854 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:38:14,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:18,061 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:38:19,409 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:23,479 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:38:24,481 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:26,855 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:38:26,856 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:38:30,046 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:30,141 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:38:35,048 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:40,221 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:38:40,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:41,854 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:38:41,855 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:38:46,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:50,300 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:38:51,350 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:53,479 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:38:56,480 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:38:57,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:38:57,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:38:57,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:39:01,376 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:02,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:07,148 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:11,459 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:12,450 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:12,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:39:12,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:39:12,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:39:17,921 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:21,536 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:23,235 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:23,479 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:39:27,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:39:27,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:39:27,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:39:28,997 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:31,617 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:34,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:39,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:41,850 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:39:41,852 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:39:41,853 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:39:41,857 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:39:42,692 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:39:42,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:39:42,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:39:42,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:39:43,697 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:44,703 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:45,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:50,920 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:53,480 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:39:53,784 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:39:56,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:39:57,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:39:57,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:39:57,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:40:01,982 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:05,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:40:07,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:12,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:13,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:40:13,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:40:13,256 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:40:15,973 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:40:18,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:23,482 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:23,483 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:40:26,045 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:40:28,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:40:28,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:40:29,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:34,479 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:38,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:40:39,557 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:43,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:40:43,253 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:40:45,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:50,278 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:40:50,589 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:53,482 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:40:56,484 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:40:58,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:40:58,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:40:58,255 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:41:02,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:02,359 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:07,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:12,453 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:12,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:13,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:41:13,254 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:41:18,470 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:23,494 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:41:23,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:24,545 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:28,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:41:28,254 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:41:29,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:33,646 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:34,713 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:39,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:43,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:41:43,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:41:43,256 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:41:45,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:45,732 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:50,189 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:53,485 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:41:55,850 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:41:56,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:41:58,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:41:58,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:41:58,255 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:42:01,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:06,443 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:07,971 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:42:11,799 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:13,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:42:13,255 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:42:17,427 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:20,129 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:42:23,003 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:23,485 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:42:28,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:42:28,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:42:28,488 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:28,489 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:42:32,207 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:42:33,991 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:38,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:42,293 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:42:43,483 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:42:43,484 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:42:44,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:49,678 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:52,355 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:42:53,486 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:42:55,487 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:42:59,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:42:59,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:42:59,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:43:01,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:02,445 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:06,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:11,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:14,526 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:14,884 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:43:14,885 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:43:17,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:22,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:23,245 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:43:23,247 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:43:23,247 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:43:23,249 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:43:23,493 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:43:23,612 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:43:24,619 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:27,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:29,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:43:29,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:43:29,887 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:43:33,134 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:36,702 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:38,400 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:43,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:44,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:43:44,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:43:44,888 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:43:47,785 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:49,179 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:53,493 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:43:54,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:57,930 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:43:59,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:43:59,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:43:59,886 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:43:59,887 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:44:05,116 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:10,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:14,885 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:44:14,886 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:44:16,110 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:21,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:23,493 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:44:26,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:30,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:44:30,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:44:30,435 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:44:31,699 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:34,168 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:44:37,210 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:42,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:45,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:44:45,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:44:45,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:44:46,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:44:47,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:52,636 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:44:53,495 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:44:56,319 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:44:58,181 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:00,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:45:00,432 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:45:00,433 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:45:03,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:06,386 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:45:09,115 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:14,117 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:15,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:45:15,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:45:15,436 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:45:18,473 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:45:19,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:23,497 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:45:25,500 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:28,543 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:45:30,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:45:30,433 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:45:30,434 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:45:30,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:35,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:38,616 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:45:41,524 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:46,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:46,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:45:46,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:45:46,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:45:50,697 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:45:52,156 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:45:53,497 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:45:57,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:00,768 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:46:01,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:46:01,785 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:46:01,786 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:46:02,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:08,001 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:10,848 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:46:13,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:16,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:46:16,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:46:16,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:46:18,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:22,958 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:46:23,497 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:46:24,498 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:29,499 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:31,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:46:31,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:46:32,065 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:46:34,941 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:39,942 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:44,943 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:46,650 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:46:46,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:46:46,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:46:46,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:46:50,929 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:53,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:46:56,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:46:56,724 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:01,068 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:01,739 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:47:01,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:47:06,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:08,810 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:11,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:16,360 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:16,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:47:16,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:47:16,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:47:18,903 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:22,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:23,499 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:47:27,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:29,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:47:29,035 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:47:29,036 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:47:29,040 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:47:30,017 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:47:31,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:31,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:47:31,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:47:32,900 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:37,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:42,115 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:43,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:46,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:47:46,740 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:47:48,952 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:52,194 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:47:53,500 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:47:54,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:47:59,503 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:01,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:48:01,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:48:01,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:48:04,303 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:48:05,040 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:10,032 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:14,381 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:48:15,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:16,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:48:16,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:48:20,951 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:23,502 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:48:26,563 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:48:26,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:31,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:48:31,741 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:48:31,950 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:37,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:38,657 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:48:42,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:46,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:48:46,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:48:46,744 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:48:47,621 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:48,738 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:48:52,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:48:53,502 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:48:58,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:00,821 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:49:01,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:49:01,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:49:03,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:08,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:10,890 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:49:14,144 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:16,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:49:16,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:49:19,955 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:22,979 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:49:23,503 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:49:25,504 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:30,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:31,741 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:49:31,742 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:49:35,980 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:39,093 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:49:41,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:46,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:46,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:49:46,743 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:49:49,168 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:49:52,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:53,504 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:49:57,505 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:49:59,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:01,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:50:01,743 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:50:02,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:07,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:10,328 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:13,837 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:18,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:50:18,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:50:19,309 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:20,397 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:23,505 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:50:24,506 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:29,507 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:32,477 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:33,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:50:33,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:50:33,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:50:35,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:40,320 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:42,546 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:46,188 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:48,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:50:48,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:50:51,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:50:53,506 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:50:54,625 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:50:56,509 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:01,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:03,129 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:51:03,130 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:51:04,701 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:51:07,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:12,316 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:13,039 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:51:13,041 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:51:13,042 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:51:13,046 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:51:13,769 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:51:14,776 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:51:18,049 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:18,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:51:18,130 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:51:18,131 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:51:23,302 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:23,512 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:51:24,840 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:51:29,111 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:33,667 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:51:33,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:51:33,669 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:51:34,841 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:36,928 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:51:39,843 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:44,845 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:47,124 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:51:48,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:51:48,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:51:48,669 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:51:49,851 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:51:53,528 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:51:55,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:00,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:03,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:52:03,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:52:03,716 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:52:05,931 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:10,932 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:14,670 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:52:16,337 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:18,668 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:52:18,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:52:18,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:52:21,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:23,507 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:52:24,764 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:52:27,508 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:32,510 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:33,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:52:33,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:52:33,670 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:52:34,834 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:52:37,828 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:42,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:44,924 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:52:48,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:48,669 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:52:48,670 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:52:48,671 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:52:53,510 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:52:53,512 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:52:57,001 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:52:58,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:03,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:03,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:53:03,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:53:03,971 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:53:07,087 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:53:09,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:14,222 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:17,153 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:53:18,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:53:18,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:53:18,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:53:20,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:23,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:53:25,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:27,224 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:53:30,860 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:34,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:53:34,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:53:34,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:53:36,176 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:37,302 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:53:41,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:46,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:49,379 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:53:49,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:53:49,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:53:49,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:53:52,187 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:53,511 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:53:57,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:53:59,456 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:02,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:04,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:54:04,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:54:04,973 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:54:07,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:09,521 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:12,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:17,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:19,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:54:19,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:54:21,599 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:23,174 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:23,512 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:54:28,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:30,708 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:33,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:34,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:54:34,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:54:34,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:54:38,764 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:40,800 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:43,765 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:48,767 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:49,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:54:49,971 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:54:50,866 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:54:53,513 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:54:54,514 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:59,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:54:59,682 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:54:59,684 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:54:59,685 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:54:59,690 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:54:59,923 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:55:00,928 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:01,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:04,693 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:04,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:55:04,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:55:09,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:11,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:14,831 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:19,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:19,971 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:55:19,972 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:55:19,972 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:55:21,095 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:23,513 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:55:25,513 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:30,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:33,172 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:34,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:55:34,971 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:55:36,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:41,151 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:43,248 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:46,190 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:50,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:55:50,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:55:50,088 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:55:51,311 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:55:53,327 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:55:53,513 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:55:56,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:01,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:03,412 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:05,087 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:56:05,088 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:56:07,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:12,272 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:15,519 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:17,704 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:20,088 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:56:20,089 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:56:20,091 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:56:23,274 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:23,514 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:56:25,588 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:28,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:33,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:35,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:56:35,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:56:35,146 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:56:37,705 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:39,387 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:44,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:47,779 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:49,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:50,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:56:50,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:56:53,517 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:56:54,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:56:56,857 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:56:59,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:04,758 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:05,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:57:05,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:57:06,972 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:57:10,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:15,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:19,075 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:57:20,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:57:20,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:57:21,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:23,516 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:57:26,518 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:29,144 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:57:31,789 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:36,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:57:36,748 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:57:36,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:57:36,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:39,220 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:57:42,901 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:47,903 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:49,298 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:57:51,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:57:51,745 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:57:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:57:53,517 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:57:58,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:01,411 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:58:04,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:06,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:58:06,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:58:06,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:58:09,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:11,500 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:58:15,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:20,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:21,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:58:21,748 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:58:23,519 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:58:23,591 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:58:25,520 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:30,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:32,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 21:58:32,257 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 21:58:32,257 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 21:58:32,260 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 21:58:32,657 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 21:58:33,666 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:58:36,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:36,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:58:36,768 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:58:41,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:46,930 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:49,987 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:58:51,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:58:51,748 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:58:51,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:53,519 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:58:57,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:58:59,137 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:02,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:07,754 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:07,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:59:07,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:59:07,960 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:59:11,244 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:13,312 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:18,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:21,311 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:22,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:59:22,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:59:22,958 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:59:23,520 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:59:23,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:28,522 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:33,432 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:34,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:37,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:59:37,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:59:37,958 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:59:39,195 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:43,497 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:44,916 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:49,918 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 21:59:52,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 21:59:52,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 21:59:52,959 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 21:59:53,521 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 21:59:53,574 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 21:59:55,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:00,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:03,658 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:05,926 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:07,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:00:07,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:00:07,959 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:00:11,120 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:13,731 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:16,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:21,949 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:22,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:00:22,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:00:22,961 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:00:23,522 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:00:25,814 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:27,523 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:32,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:35,877 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:37,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:00:37,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:00:37,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:37,960 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:00:43,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:47,969 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:48,262 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:52,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:00:52,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:00:52,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:00:53,523 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:00:53,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:00:58,043 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:00:58,934 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:03,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:07,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:01:07,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:01:07,960 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:01:08,116 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:01:09,216 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:14,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:18,198 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:01:19,362 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:22,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:01:22,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:01:22,960 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:01:23,523 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:01:24,525 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:29,286 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:01:30,435 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:35,436 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:37,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:01:37,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:01:37,961 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:01:39,364 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:01:41,253 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:46,254 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:51,472 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:01:52,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:01:52,959 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:01:52,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:01:52,961 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:01:53,524 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:01:57,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:01,590 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:02:02,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:07,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:07,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:02:07,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:02:07,965 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:02:11,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:02:11,517 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:02:11,518 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:02:11,520 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:02:11,661 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:02:12,715 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:13,675 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:02:15,685 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:02:18,695 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:22,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:02:22,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:02:22,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:02:23,525 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:02:24,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:29,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:34,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:37,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:02:37,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:02:37,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:02:40,225 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:45,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:50,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:02:52,960 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:02:52,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:02:52,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:02:53,525 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:02:55,526 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:00,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:05,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:07,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:03:07,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:03:07,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:03:11,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:16,200 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:21,201 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:22,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:03:22,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:03:22,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:03:23,526 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:03:26,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:31,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:36,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:37,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:03:37,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:03:37,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:03:42,141 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:47,142 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:52,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:03:52,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:03:52,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:03:52,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:03:53,528 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:03:57,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:02,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:07,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:07,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:04:07,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:04:07,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:04:13,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:18,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:22,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:04:22,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:04:22,963 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:04:23,147 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:23,529 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:04:28,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:33,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:37,961 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:04:37,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:04:37,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:04:39,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:44,213 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:49,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:52,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:04:52,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:04:52,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:04:53,530 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:04:54,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:04:59,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:04,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:07,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:05:07,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:05:07,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:05:10,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:15,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:20,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:22,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:05:22,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:05:22,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:05:23,529 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:05:25,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:30,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:35,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:37,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:05:37,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:05:37,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:05:41,213 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:46,214 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:51,215 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:05:52,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:05:52,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:05:52,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:05:53,530 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:05:56,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:01,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:06,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:07,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:06:07,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:06:07,964 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:06:12,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:17,220 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:22,221 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:22,962 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:06:22,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:06:22,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:06:23,530 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:06:27,531 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:32,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:37,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:37,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:06:37,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:06:37,965 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:06:43,127 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:48,128 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:52,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:06:52,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:06:52,965 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:06:53,531 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:06:53,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:06:58,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:03,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:07,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:07:07,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:07:07,966 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:07:09,183 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:14,184 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:19,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:22,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:07:22,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:07:22,965 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:07:23,532 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:07:24,532 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:29,533 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:34,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:37,963 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:07:37,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:07:37,966 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:07:40,204 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:45,207 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:50,208 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:07:52,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:07:52,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:07:52,966 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:07:53,532 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:07:55,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:00,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:05,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:07,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:08:07,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:08:07,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:08:11,161 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:16,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:21,162 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:22,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:08:22,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:08:22,966 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:08:23,533 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:08:26,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:31,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:36,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:37,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:08:37,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:08:37,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:08:42,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:47,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:52,219 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:08:52,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:08:52,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:08:52,966 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:08:53,533 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:08:57,534 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:02,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:07,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:07,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:09:07,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:09:07,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:09:13,135 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:18,136 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:22,964 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:09:22,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:09:22,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:09:23,226 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:23,534 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:09:28,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:33,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:37,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:09:37,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:09:37,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:09:39,196 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:44,197 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:49,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:52,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:09:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:09:52,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:09:53,534 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:09:54,535 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:09:59,536 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:04,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:07,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:10:07,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:10:07,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:10:10,237 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:15,238 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:20,239 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:22,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:10:22,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:10:22,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:10:23,536 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:10:25,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:30,538 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:35,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:37,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:10:37,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:10:37,967 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:10:41,145 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:46,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:51,146 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:10:52,965 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:10:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:10:52,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:10:53,537 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:10:56,537 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:01,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:06,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:07,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:11:07,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:11:07,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:11:12,193 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:17,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:22,194 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:22,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:11:22,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:11:22,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:11:23,537 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:11:27,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:32,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:37,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:37,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:11:37,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:11:37,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:11:43,229 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:48,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:52,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:11:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:11:52,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:11:53,538 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:11:53,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:11:58,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:03,540 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:07,966 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:12:07,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:12:07,968 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:12:09,198 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:14,199 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:19,200 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:22,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:12:22,970 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:12:22,971 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:12:23,538 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:12:24,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:29,547 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:34,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:37,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:12:37,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:12:37,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:12:40,152 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:45,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:50,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:12:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:12:52,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:12:52,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:12:53,539 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:12:55,539 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:00,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:05,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:07,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:13:07,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:13:07,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:13:11,264 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:16,265 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:21,266 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:22,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:13:22,968 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:13:22,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:13:23,540 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:13:26,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:31,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:34,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:13:34,062 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:13:34,062 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:13:34,065 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:13:34,446 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:13:36,458 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:13:37,067 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:37,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:13:37,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:13:37,969 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:13:42,153 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:47,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:52,155 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:13:52,967 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:13:52,969 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:13:52,970 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:13:53,540 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:13:57,541 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:02,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:04,663 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:14:08,298 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:14:08,367 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:09,486 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:14:09,487 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:14:13,742 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:18,743 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:23,428 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:14:23,464 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:14:23,465 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:14:23,708 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:14:24,339 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:29,340 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:34,341 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:38,635 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:14:38,685 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:14:38,686 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:14:40,043 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:45,044 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:50,707 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:50,904 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:14:53,541 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:14:54,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:14:56,542 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:14:58,200 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:14:58,200 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:15:00,954 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:15:02,457 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:05,638 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:15:07,458 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:12,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:13,418 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:15:17,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:20,990 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:15:20,990 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:15:23,086 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:15:23,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:23,541 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:15:28,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:15:28,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:28,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:15:33,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:38,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:43,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:15:43,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:15:43,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:48,649 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:53,542 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:15:54,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:15:58,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:15:58,419 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:15:59,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:04,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:09,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:13,420 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:16:13,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:16:14,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:19,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:23,542 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:16:25,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:28,419 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:16:28,420 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:16:30,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:32,440 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:16:36,023 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:41,024 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:44,384 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:16:44,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:16:44,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:16:46,535 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:16:46,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:51,908 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:53,544 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:16:57,527 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:16:58,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:16:59,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:16:59,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:17:02,576 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:07,577 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:10,719 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:17:12,639 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:14,383 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:17:14,384 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:17:18,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:22,807 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:17:23,544 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:17:24,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:29,385 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:17:29,386 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:17:29,387 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:17:29,655 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:34,885 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:17:34,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:39,999 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:44,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:17:44,495 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:17:44,496 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:17:45,722 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:46,971 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:17:50,905 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:53,545 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:17:56,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:17:59,052 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:17:59,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:17:59,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:17:59,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:18:01,663 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:06,664 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:11,134 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:18:12,053 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:14,493 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:18:14,494 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:18:14,495 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:18:17,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:22,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:23,240 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:18:23,546 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:18:28,548 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:30,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:18:30,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:18:30,703 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:18:33,318 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:18:33,947 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:38,948 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:44,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:45,441 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:18:45,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:18:45,701 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:18:49,907 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:53,547 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:18:55,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:18:57,550 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:00,700 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:19:00,701 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:19:00,701 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:19:00,902 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:05,904 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:08,628 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:11,634 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:16,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:19:16,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:19:16,394 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:19:17,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:18,702 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:23,051 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:23,548 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:19:28,549 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:30,793 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:31,391 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:19:31,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:19:31,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:19:33,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:38,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:42,891 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:44,021 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:46,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:19:46,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:19:49,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:19:53,549 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:19:55,002 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:19:55,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:00,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:01,393 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:20:01,394 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:20:01,396 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:20:06,255 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:07,091 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:20:11,256 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:16,258 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:16,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:20:16,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:20:19,204 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:20:21,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:23,550 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:20:26,551 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:31,392 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:20:31,393 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:20:31,609 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:36,466 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:20:36,469 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:20:36,470 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:20:36,473 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:20:37,336 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:20:37,337 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:20:37,475 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:39,348 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:20:42,476 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:46,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:20:46,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:20:46,462 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:20:47,672 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:49,432 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:20:53,121 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:20:53,550 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:20:58,553 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:01,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:21:01,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:21:01,461 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:21:01,571 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:03,654 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:08,656 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:13,679 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:14,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:16,460 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:21:16,461 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:21:16,464 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:21:19,723 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:23,551 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:21:25,552 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:25,759 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:30,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:32,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:21:32,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:21:32,358 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:21:34,853 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:35,957 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:40,958 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:46,059 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:46,939 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:47,356 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:21:47,357 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:21:51,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:21:53,553 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:21:57,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:21:57,149 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:02,150 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:02,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:22:02,357 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:22:02,358 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:22:07,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:09,147 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:22:13,154 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:17,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:22:17,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:22:17,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:22:19,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:21,233 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:22:23,553 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:22:24,554 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:29,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:31,311 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:22:32,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:22:32,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:22:35,071 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:40,073 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:41,391 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:22:45,629 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:47,874 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:22:47,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:22:51,070 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:22:53,555 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:22:55,714 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:22:56,556 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:01,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:02,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:23:02,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:23:06,751 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:07,837 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:23:11,753 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:17,729 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:17,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:23:17,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:23:17,909 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:23:23,126 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:23,557 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:23:26,975 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:23:28,687 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:32,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:23:32,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:23:34,133 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:39,065 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:23:39,866 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:44,867 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:47,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:23:47,878 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:23:47,879 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:23:50,218 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:23:51,161 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:23:53,557 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:23:55,558 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:01,301 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:24:01,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:02,875 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:24:02,876 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:24:07,131 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:13,124 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:13,393 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:24:17,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:24:17,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:24:19,052 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:23,558 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:24:23,569 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:24:23,570 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:24:23,572 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:24:23,576 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:24:24,483 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:24:24,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:25,489 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:24:29,580 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:32,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:24:32,878 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:24:35,211 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:35,558 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:24:40,212 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:46,074 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:47,696 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:24:47,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:24:47,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:24:52,058 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:53,560 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:24:57,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:24:59,784 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:02,366 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:02,876 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:25:02,877 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:25:08,143 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:09,856 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:13,809 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:18,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:25:18,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:25:18,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:25:19,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:21,936 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:23,560 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:25:24,563 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:29,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:32,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:33,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:25:33,096 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:25:35,353 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:40,355 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:44,110 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:45,676 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:48,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:25:48,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:25:51,358 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:25:53,172 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:25:53,561 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:25:56,562 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:01,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:03,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:26:03,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:26:05,258 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:26:07,543 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:12,545 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:15,347 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:26:17,662 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:18,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:26:18,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:26:23,267 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:23,576 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:26:27,452 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:26:28,928 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:33,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:26:33,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:26:34,365 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:39,586 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:26:40,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:45,301 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:48,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:26:48,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:26:49,660 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:26:50,403 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:26:53,563 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:26:55,564 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:00,696 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:01,774 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:03,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:27:03,097 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:27:06,318 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:11,604 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:11,839 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:16,613 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:18,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:27:18,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:27:22,252 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:23,564 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:27:23,933 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:27,565 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:33,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:27:33,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:27:33,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:34,027 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:38,315 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:43,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:44,101 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:48,097 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:27:48,098 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:27:49,346 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:53,565 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:27:54,895 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:27:56,302 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:27:59,896 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:03,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:28:03,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:28:05,299 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:06,731 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:28:06,733 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:28:06,733 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:28:06,735 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:28:07,440 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:28:07,441 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:28:10,738 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:15,740 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:18,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:28:18,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:28:19,544 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:28:21,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:23,565 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:28:26,567 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:29,620 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:28:31,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:33,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:28:33,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:28:37,319 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:41,739 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:28:42,455 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:47,456 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:48,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:28:48,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:28:51,799 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:28:53,242 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:28:53,567 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:28:58,568 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:03,098 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:29:03,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:29:03,889 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:29:04,313 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:09,314 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:13,960 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:29:14,832 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:18,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:29:18,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:29:20,275 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:23,568 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:29:25,622 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:26,037 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:29:30,623 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:33,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:29:33,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:29:36,096 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:36,118 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:29:41,107 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:46,191 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:29:46,782 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:48,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:29:48,099 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:29:52,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:53,570 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:29:57,803 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:29:58,271 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:02,804 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:03,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:30:03,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:30:08,339 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:08,611 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:13,612 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:18,099 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:30:18,100 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:30:19,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:20,434 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:23,569 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:30:24,570 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:29,993 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:30,530 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:33,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:30:33,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:30:35,317 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:40,325 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:41,631 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:45,983 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:48,100 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:30:48,101 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:30:51,291 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:30:53,570 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:30:53,728 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:30:56,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:01,574 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:03,103 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:31:03,104 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:31:03,104 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:31:05,851 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:31:07,516 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:12,519 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:17,521 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:18,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:31:18,102 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:31:19,955 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:31:23,348 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:23,570 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:31:28,572 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:32,380 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:31:33,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:31:33,102 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:31:34,344 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:39,345 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:44,474 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:31:44,735 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:48,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:31:48,102 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:31:50,326 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:53,572 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:31:53,914 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:31:53,916 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:31:53,917 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:31:53,920 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:31:54,555 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:31:55,561 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:31:55,922 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:31:56,566 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:00,923 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:03,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:32:03,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:32:03,748 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:32:06,227 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:06,670 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:11,228 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:17,028 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:17,747 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:18,745 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:32:18,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:32:18,747 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:32:22,981 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:23,572 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:32:27,832 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:28,462 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:33,463 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:33,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:32:33,746 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:32:33,747 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:32:38,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:39,935 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:44,101 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:48,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:32:48,749 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:32:48,749 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:32:49,925 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:32:52,010 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:32:53,572 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:32:55,573 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:00,575 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:02,087 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:33:03,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:33:03,748 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:33:05,992 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:10,998 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:14,250 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:33:16,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:18,747 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:33:18,748 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:33:21,936 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:23,580 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:33:26,373 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:33:26,943 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:31,945 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:35,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:33:35,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:33:35,285 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:33:37,472 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:38,574 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:33:42,736 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:47,737 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:50,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:33:50,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:33:52,240 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:33:53,230 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:33:53,577 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:33:58,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:02,324 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:34:04,379 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:05,279 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:34:05,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:34:05,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:34:09,447 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:14,429 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:34:14,590 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:19,591 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:20,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:34:20,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:34:20,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:34:23,577 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:34:24,943 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:26,586 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:34:29,944 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:34,946 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:35,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:34:35,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:34:35,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:34:38,708 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:34:40,138 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:45,140 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:50,280 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:34:50,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:34:50,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:34:50,871 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:34:53,577 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:34:55,578 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:01,374 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:02,022 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:35:05,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:35:05,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:35:06,528 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:11,530 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:14,253 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:35:17,041 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:20,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:35:20,282 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:35:20,284 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:35:22,515 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:23,578 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:35:26,346 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:35:27,973 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:32,977 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:35,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:35:35,281 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:35:36,439 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:35:38,501 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:43,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:48,502 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:50,281 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:35:50,282 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:35:53,529 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:35:53,579 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:35:58,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:03,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:05,389 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:36:05,404 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:36:05,424 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:36:08,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: partial_history 2023-11-22 22:36:08,408 DEBUG SenderThread:2219461 [sender.py:send():380] send: history 2023-11-22 22:36:08,409 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: summary_record 2023-11-22 22:36:08,413 INFO SenderThread:2219461 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-22 22:36:08,637 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/wandb-summary.json 2023-11-22 22:36:09,415 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:10,648 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:36:14,416 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:19,417 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:20,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:36:20,414 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:36:22,737 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:36:23,580 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:36:24,581 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:29,582 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:34,666 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:34,821 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:36:35,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:36:35,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:36:40,588 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:45,830 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:46,927 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:36:50,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:36:50,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:36:51,598 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:53,593 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:36:57,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:36:59,016 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:37:02,185 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:05,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:37:05,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:37:07,579 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:11,099 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:37:13,406 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:18,411 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:20,399 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:37:20,400 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:37:22,382 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:37:23,582 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:37:23,583 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:28,584 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:33,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:35,398 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:37:35,399 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:37:38,631 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:42,512 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:37:44,177 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:49,178 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:52,288 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:37:52,293 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:37:53,583 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:37:54,269 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:37:56,655 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:37:59,270 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:04,271 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:07,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:38:07,294 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:38:07,295 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:38:09,054 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:38:09,819 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:14,811 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:19,813 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:22,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:38:22,293 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:38:22,294 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status 2023-11-22 22:38:23,584 DEBUG SenderThread:2219461 [sender.py:send():380] send: stats 2023-11-22 22:38:24,368 INFO Thread-12 :2219461 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231122_100021-qrxr9o47/files/output.log 2023-11-22 22:38:25,585 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:30,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:35,594 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: status_report 2023-11-22 22:38:37,289 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: keepalive 2023-11-22 22:38:37,300 DEBUG HandlerThread:2219461 [handler.py:handle_request():146] handle_request: stop_status 2023-11-22 22:38:37,310 DEBUG SenderThread:2219461 [sender.py:send_request():407] send_request: stop_status