2024-04-29 07:27:05,569 INFO StreamThr :108 [internal.py:wandb_internal():86] W&B internal server running at pid: 108, started at: 2024-04-29 07:27:05.569057 2024-04-29 07:27:05,571 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status 2024-04-29 07:27:06,061 INFO WriterThread:108 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/run-0t7z1bit.wandb 2024-04-29 07:27:06,061 DEBUG SenderThread:108 [sender.py:send():378] send: header 2024-04-29 07:27:06,064 DEBUG SenderThread:108 [sender.py:send():378] send: run 2024-04-29 07:27:06,242 INFO SenderThread:108 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files 2024-04-29 07:27:06,242 INFO SenderThread:108 [sender.py:_start_run_threads():1099] run started: 0t7z1bit with start time 1714375625.570451 2024-04-29 07:27:06,249 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: check_version 2024-04-29 07:27:06,250 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: check_version 2024-04-29 07:27:06,337 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: run_start 2024-04-29 07:27:06,348 DEBUG HandlerThread:108 [system_info.py:__init__():26] System info init 2024-04-29 07:27:06,348 DEBUG HandlerThread:108 [system_info.py:__init__():41] System info init done 2024-04-29 07:27:06,348 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 07:27:06,348 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 07:27:06,348 INFO HandlerThread:108 [system_monitor.py:probe():214] Collecting system info 2024-04-29 07:27:06,349 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 07:27:06,349 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 07:27:06,350 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 07:27:06,351 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 07:27:06,352 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 07:27:06,362 DEBUG HandlerThread:108 [system_info.py:probe():150] Probing system 2024-04-29 07:27:06,364 DEBUG HandlerThread:108 [gitlib.py:_init_repo():56] git repository is invalid 2024-04-29 07:27:06,364 DEBUG HandlerThread:108 [system_info.py:probe():198] Probing system done 2024-04-29 07:27:06,364 DEBUG HandlerThread:108 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-29T07:27:06.362332', 'startedAt': '2024-04-29T07:27:05.562699', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'be592b10bdc8', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.132, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.132, 'min': 0.0, 'max': 0.0}, {'current': 2000.132, 'min': 0.0, 'max': 0.0}, {'current': 2000.132, 'min': 0.0, 'max': 0.0}, {'current': 2000.132, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5598.571662902832}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} 2024-04-29 07:27:06,364 INFO HandlerThread:108 [system_monitor.py:probe():224] Finished collecting system info 2024-04-29 07:27:06,364 INFO HandlerThread:108 [system_monitor.py:probe():227] Publishing system info 2024-04-29 07:27:06,364 DEBUG HandlerThread:108 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-04-29 07:27:07,244 INFO Thread-12 :108 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/conda-environment.yaml 2024-04-29 07:28:03,259 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/conda-environment.yaml 2024-04-29 07:28:03,475 DEBUG HandlerThread:108 [system_info.py:_save_conda():219] Saving conda packages done 2024-04-29 07:28:03,475 INFO HandlerThread:108 [system_monitor.py:probe():229] Finished publishing system info 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: python_packages 2024-04-29 07:28:03,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: stop_status 2024-04-29 07:28:03,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-29 07:28:03,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:03,491 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:03,492 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: python_packages 2024-04-29 07:28:03,497 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: stop_status 2024-04-29 07:28:03,502 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:03,546 DEBUG SenderThread:108 [sender.py:send():378] send: telemetry 2024-04-29 07:28:03,546 DEBUG SenderThread:108 [sender.py:send():378] send: config 2024-04-29 07:28:03,548 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,548 DEBUG SenderThread:108 [sender.py:send():378] send: telemetry 2024-04-29 07:28:03,548 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,548 WARNING SenderThread:108 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) 2024-04-29 07:28:03,548 DEBUG SenderThread:108 [sender.py:send():378] send: telemetry 2024-04-29 07:28:03,549 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,549 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,550 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,550 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,550 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:03,550 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:03,551 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,552 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,553 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,553 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 07:28:03,553 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:03,553 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:03,553 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:03,554 DEBUG SenderThread:108 [sender.py:send():378] send: files 2024-04-29 07:28:03,554 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now 2024-04-29 07:28:03,746 INFO wandb-upload_0:108 [upload_job.py:push():131] Uploaded file /tmp/tmp7rkmxf57wandb/dls4elmd-wandb-metadata.json 2024-04-29 07:28:04,259 INFO Thread-12 :108 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:04,260 INFO Thread-12 :108 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-metadata.json 2024-04-29 07:28:04,260 INFO Thread-12 :108 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:28:04,260 INFO Thread-12 :108 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/requirements.txt 2024-04-29 07:28:04,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:04,375 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:04,375 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:04,376 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:05,260 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:06,260 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:28:06,353 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 07:28:06,354 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:28:06,360 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:07,261 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/config.yaml 2024-04-29 07:28:08,504 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:11,469 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:13,507 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:16,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:18,508 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:18,690 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:18,691 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:18,691 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:18,692 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:19,266 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:20,266 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:28:21,752 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:22,786 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:22,788 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:22,788 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:22,788 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:23,268 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:23,509 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:26,789 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:28,518 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:31,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:33,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:36,356 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:28:37,356 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:37,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:37,620 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:37,621 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:37,621 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:38,274 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:38,526 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:40,275 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:28:41,745 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:41,746 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:41,746 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:41,747 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:42,275 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:42,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:43,537 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:47,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:48,548 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:52,749 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:53,552 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:28:56,322 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:28:56,323 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:28:56,324 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:28:56,324 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:28:57,281 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:28:58,281 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:28:58,378 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:28:58,557 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:00,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:00,471 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:00,471 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:00,472 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:01,282 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:03,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:03,558 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:06,356 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:29:08,564 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:09,357 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:13,570 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:14,358 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:15,175 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:15,177 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:15,177 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:15,178 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:15,288 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:18,289 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:29:18,571 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:19,352 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:19,353 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:19,353 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:19,354 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:20,290 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:20,354 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:23,578 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:25,355 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:28,579 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:30,356 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:33,580 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:34,184 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:34,185 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:34,186 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:34,187 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:34,295 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:36,238 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:36,296 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:29:36,357 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:29:38,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:38,390 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:38,390 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:38,391 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:38,581 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:39,297 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:41,391 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:43,585 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:46,392 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:48,591 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:51,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:53,593 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:29:53,809 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:53,811 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:53,811 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:53,812 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:54,308 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:56,308 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:29:56,865 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:29:58,053 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:29:58,054 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:29:58,054 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:29:58,055 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:29:58,309 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:29:58,594 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:02,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:03,597 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:06,358 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:30:07,359 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:08,600 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:12,360 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:13,295 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:13,296 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:13,297 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:13,297 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:13,315 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:13,607 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:16,316 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:30:17,567 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:17,568 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:17,568 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:17,569 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:17,571 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:18,317 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:18,610 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:22,572 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:23,614 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:27,573 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:28,615 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:32,573 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:32,943 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:32,944 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:32,945 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:32,947 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:33,323 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:33,617 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:34,323 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:30:36,359 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:30:37,234 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:37,235 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:37,235 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:37,236 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:37,325 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:38,236 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:38,618 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:43,237 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:43,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:48,238 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:48,630 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:53,022 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:53,023 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:53,024 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:53,024 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:53,330 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:53,632 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:54,090 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:30:54,330 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:30:57,338 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:30:57,339 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:30:57,340 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:30:57,342 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:30:58,333 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:30:58,638 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:30:59,343 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:03,644 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:04,344 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:06,360 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:31:08,645 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:09,361 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:12,903 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:12,904 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:12,905 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:12,907 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:13,338 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:13,646 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:14,338 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:31:14,959 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:17,239 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:17,241 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:17,241 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:17,241 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:17,340 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:18,650 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:20,242 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:23,651 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:25,243 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:28,652 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:30,244 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:32,885 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:32,886 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:32,886 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:32,887 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:33,346 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:33,661 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:34,346 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:31:35,950 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:36,361 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:31:37,229 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:37,231 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:37,231 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:37,231 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:37,347 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:38,662 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:41,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:43,663 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:46,233 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:48,664 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:51,234 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:53,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:53,004 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:53,005 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:53,005 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:53,353 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:53,664 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:31:54,353 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:31:57,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:31:57,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:31:57,371 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:31:57,371 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:31:57,373 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:31:58,355 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:31:58,665 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:02,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:03,957 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:06,362 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:32:08,363 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:08,969 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:13,364 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:13,570 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:13,571 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:13,572 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:13,572 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:13,991 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:14,361 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:32:16,361 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:32:17,949 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:17,951 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:17,951 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:17,951 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:18,362 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:32:18,952 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:18,994 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:23,953 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:24,019 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:28,954 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:29,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:33,849 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:33,851 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:33,851 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:33,854 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:34,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:34,368 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:32:34,906 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:36,363 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:32:36,369 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:32:38,247 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:38,249 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:38,249 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:38,249 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:38,370 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:32:39,064 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:40,250 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:44,065 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:45,251 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:49,082 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:50,252 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:54,083 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:54,250 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:54,251 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:54,252 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:54,252 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:54,376 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:32:55,305 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:32:56,377 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:32:58,667 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:32:58,668 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:32:58,668 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:32:58,669 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:32:59,084 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:32:59,378 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:00,669 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:04,136 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:05,670 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:06,369 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:33:09,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:11,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:14,145 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:15,045 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:33:15,046 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:33:15,046 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:33:15,047 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:33:15,384 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:16,385 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:33:17,101 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:19,146 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:19,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:33:19,472 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:33:19,472 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:33:19,472 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:33:20,386 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:22,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:24,147 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:27,474 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:29,153 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:32,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:34,154 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:35,535 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:33:35,537 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:33:35,537 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:33:35,537 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:33:36,365 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:33:36,392 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:38,365 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:38,393 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:33:39,155 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:39,967 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:33:39,968 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:33:39,968 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:33:39,969 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:33:40,394 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:43,969 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:44,162 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:48,971 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:49,163 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:53,972 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:54,166 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:33:55,993 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:33:55,995 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:33:55,995 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:33:55,996 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:33:56,400 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:33:58,401 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:33:59,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:33:59,168 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:00,413 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:00,414 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:00,415 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:00,415 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:01,402 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:04,174 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:04,416 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:06,366 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:34:09,175 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:10,367 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:14,181 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:15,368 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:16,464 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:16,466 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:16,466 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:16,469 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:17,407 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:18,408 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:34:19,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:20,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:20,901 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:20,902 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:20,902 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:20,902 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:21,409 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:24,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:25,903 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:29,195 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:30,904 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:34,201 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:35,905 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:36,366 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:34:37,327 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:37,328 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:37,329 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:37,329 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:37,415 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:38,415 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:34:39,202 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:41,382 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:41,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:41,763 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:41,763 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:41,763 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:42,417 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:44,203 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:46,764 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:49,209 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:51,765 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:54,213 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:34:56,766 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:34:57,857 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:34:57,858 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:34:57,859 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:34:57,861 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:34:58,425 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:34:59,219 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:00,425 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:35:01,916 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:02,275 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:02,276 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:02,277 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:02,277 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:35:02,426 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:35:04,223 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:06,367 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:35:07,368 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:09,227 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:12,369 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:14,228 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:17,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:18,327 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:18,328 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:18,328 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:18,329 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:35:18,432 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:35:19,231 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:20,433 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:35:22,383 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:22,758 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:22,759 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:22,760 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:22,760 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:35:23,434 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:35:24,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:27,761 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:29,234 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:32,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:34,241 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:36,369 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:35:38,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:39,158 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:39,159 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:39,160 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:39,160 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:35:39,242 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:39,440 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:35:40,441 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:35:43,589 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:43,590 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:43,590 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:43,591 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:43,591 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:35:44,243 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:44,442 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:35:48,592 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:49,244 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:53,593 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:54,251 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:58,594 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:35:59,252 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:35:59,651 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:35:59,652 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:35:59,653 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:35:59,653 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:00,448 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:02,449 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:36:03,727 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:04,086 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:36:04,087 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:36:04,087 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:36:04,088 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:04,253 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:04,450 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:06,370 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:36:09,254 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:09,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:14,260 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:14,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:19,261 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:19,372 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:20,177 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:36:20,179 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:36:20,179 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:36:20,180 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:20,455 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:22,456 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:36:24,263 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:24,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:36:24,615 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:36:24,615 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:36:24,616 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:24,616 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:25,458 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:29,263 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:29,617 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:34,271 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:34,618 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:36,371 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:36:39,272 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:40,372 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:40,671 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:36:40,673 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:36:40,673 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:36:40,674 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:41,464 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:42,464 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:36:44,274 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:45,094 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:36:45,095 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:36:45,095 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:36:45,096 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:36:45,466 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:36:46,096 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:49,582 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:51,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:54,596 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:36:56,098 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:36:59,597 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:01,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:01,529 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:01,530 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:01,531 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:01,531 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:02,472 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:02,472 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:37:04,603 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:05,957 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:05,958 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:05,958 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:05,959 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:06,372 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:37:06,372 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:06,473 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:09,604 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:11,373 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:14,610 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:16,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:19,611 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:21,376 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:22,061 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:22,062 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:22,063 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:22,063 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:22,479 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:24,480 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:37:24,612 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:26,502 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:26,503 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:26,503 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:26,504 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:26,504 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:27,482 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:29,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:31,505 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:34,618 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:36,372 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:37:37,373 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:39,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:42,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:42,567 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:42,568 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:42,569 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:42,569 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:43,487 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:44,488 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:37:44,625 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:46,988 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:37:46,989 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:37:46,989 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:37:46,990 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:37:47,489 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:37:47,990 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:49,627 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:52,991 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:54,789 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:37:57,992 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:37:59,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:02,993 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:03,345 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:03,346 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:03,347 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:03,347 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:03,495 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:04,495 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:38:04,791 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:06,374 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:38:07,754 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:07,755 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:07,755 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:07,755 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:08,497 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:08,756 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:09,798 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:13,757 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:14,800 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:18,758 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:19,806 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:23,759 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:23,799 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:23,801 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:23,801 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:23,801 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:24,503 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:24,814 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:26,504 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:38:28,229 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:28,230 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:28,230 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:28,230 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:28,505 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:29,231 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:29,816 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:34,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:34,822 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:36,374 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:38:39,375 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:39,829 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:44,341 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:44,343 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:44,343 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:44,343 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:44,397 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:44,511 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:44,835 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:46,512 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:38:48,786 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:38:48,787 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:38:48,788 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:38:48,788 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:38:49,513 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:38:49,789 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:49,841 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:54,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:54,846 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:38:59,791 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:38:59,850 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:04,792 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:04,851 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:04,874 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:04,876 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:04,877 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:04,879 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:05,519 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:06,375 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:39:06,519 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:39:09,318 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:09,319 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:09,319 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:09,320 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:09,521 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:09,854 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:10,320 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:14,978 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:15,321 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:19,990 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:20,322 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:24,991 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:25,323 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:25,728 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:25,730 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:25,730 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:25,730 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:26,527 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:26,528 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:39:29,992 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:30,154 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:30,155 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:30,155 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:30,156 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:30,529 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:31,156 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:34,993 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:36,157 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:36,376 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:39:40,000 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:41,378 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:45,000 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:46,243 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:46,244 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:46,245 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:46,245 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:46,535 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:47,298 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:48,535 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:39:50,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:50,683 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:39:50,684 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:39:50,685 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:39:50,685 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:39:51,537 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:39:52,686 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:39:55,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:39:57,687 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:00,004 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:02,688 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:05,005 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:06,382 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:40:06,764 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:06,766 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:06,766 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:06,766 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:07,542 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:07,833 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:08,543 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:40:10,012 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:11,197 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:11,198 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:11,199 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:11,199 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:11,544 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:13,200 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:15,013 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:18,201 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:20,014 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:23,202 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:25,015 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:27,604 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:27,605 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:27,606 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:27,606 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:28,550 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:28,550 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:40:28,659 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:30,022 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:32,029 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:32,030 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:32,030 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:32,031 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:32,551 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:34,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:35,022 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:36,378 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:40:39,379 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:40,029 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:44,380 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:45,036 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:48,087 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:48,088 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:48,089 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:48,091 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:48,557 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:50,037 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:50,143 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:40:50,558 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:40:52,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:40:52,520 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:40:52,520 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:40:52,521 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:40:52,559 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:40:55,039 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:40:55,522 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:00,040 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:00,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:05,046 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:05,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:06,379 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:41:08,585 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:08,587 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:08,587 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:08,587 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:09,565 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:10,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:10,566 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:41:10,642 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:13,027 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:13,028 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:13,028 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:13,029 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:13,567 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:15,054 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:16,030 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:20,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:21,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:25,057 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:26,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:29,096 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:29,097 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:29,097 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:29,099 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:29,573 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:30,058 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:30,573 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:41:31,154 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:33,537 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:33,538 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:33,538 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:33,539 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:33,575 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:35,060 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:36,380 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:41:36,381 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:40,060 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:41,381 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:45,066 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:46,383 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:49,981 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:49,982 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:49,983 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:49,983 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:50,067 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:50,581 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:52,040 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:41:52,582 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:41:54,424 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:41:54,425 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:41:54,425 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:41:54,426 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:41:54,583 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:41:55,068 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:41:57,427 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:00,074 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:02,428 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:05,085 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:06,381 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:42:08,382 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:10,092 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:10,499 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:10,501 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:10,501 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:10,501 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:10,589 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:42:12,589 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:42:13,557 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:14,932 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:14,934 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:14,934 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:14,935 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:15,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:15,591 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:42:18,936 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:20,096 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:23,936 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:25,102 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:28,937 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:30,103 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:30,956 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:30,957 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:30,958 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:30,958 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:31,625 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:42:32,625 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:42:34,023 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:35,104 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:35,379 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:35,380 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:35,380 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:35,380 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:35,626 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:42:36,382 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:42:39,383 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:40,110 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:44,384 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:45,111 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:49,385 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:50,118 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:51,781 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:51,782 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:51,783 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:51,783 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:52,633 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:42:52,633 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:42:54,842 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:42:55,119 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:42:56,205 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:42:56,206 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:42:56,206 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:42:56,207 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:42:56,635 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:00,123 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:00,208 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:05,124 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:05,208 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:06,383 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:43:10,130 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:10,384 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:12,235 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:12,236 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:12,237 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:12,237 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:12,640 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:14,641 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:43:15,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:16,290 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:16,675 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:16,676 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:16,676 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:16,677 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:17,642 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:20,145 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:21,678 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:25,151 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:26,679 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:30,152 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:31,679 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:32,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:32,763 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:32,763 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:32,764 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:33,648 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:34,648 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:43:35,159 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:36,384 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:43:37,194 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:37,195 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:37,196 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:37,196 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:37,196 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:37,650 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:40,160 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:42,197 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:45,167 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:47,198 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:50,169 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:52,199 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:53,282 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:53,283 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:53,284 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:53,284 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:53,655 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:43:54,656 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:43:55,178 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:43:57,337 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:43:57,704 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:43:57,705 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:43:57,705 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:43:57,706 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:43:58,657 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:00,179 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:02,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:05,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:06,384 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:44:08,385 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:10,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:13,386 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:14,110 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:44:14,111 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:44:14,112 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:44:14,112 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:44:14,663 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:14,663 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:44:15,193 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:18,546 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:44:18,547 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:44:18,547 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:44:18,547 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:18,548 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:44:18,664 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:20,194 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:23,549 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:25,200 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:28,550 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:30,210 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:33,551 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:34,623 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:44:34,625 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:44:34,625 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:44:34,626 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:44:34,670 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:35,460 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:36,385 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:44:38,749 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:39,672 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:44:40,469 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:41,150 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:44:41,151 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:44:41,151 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:44:41,151 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:44:41,673 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:44,152 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:45,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:49,153 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:50,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:54,154 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:55,482 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:44:57,259 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:44:57,260 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:44:57,261 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:44:57,261 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:44:57,679 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:44:59,314 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:44:59,679 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:45:00,486 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:01,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:01,711 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:01,711 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:01,712 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:02,681 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:45:04,712 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:05,684 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:06,386 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:45:10,387 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:10,692 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:15,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:15,695 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:18,174 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:18,175 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:18,175 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:18,176 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:18,686 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:45:19,687 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:45:20,695 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:21,231 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:22,616 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:22,617 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:22,617 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:22,618 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:22,688 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:45:25,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:26,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:30,714 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:31,620 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:35,715 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:36,387 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:45:37,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:38,718 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:38,720 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:38,720 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:38,721 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:39,695 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:45:40,716 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:41,695 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:45:42,780 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:43,160 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:43,161 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:43,161 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:43,162 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:43,696 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:45:45,730 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:48,163 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:50,732 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:53,164 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:55,737 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:45:58,165 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:45:59,278 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:45:59,279 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:45:59,280 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:45:59,280 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:45:59,702 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:00,739 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:01,702 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:46:03,339 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:03,714 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:46:03,715 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:46:03,716 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:46:03,716 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:46:04,704 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:05,746 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:06,388 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:46:08,389 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:11,023 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:13,390 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:16,024 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:18,391 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:20,153 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:46:20,155 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:46:20,155 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:46:20,156 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:46:20,710 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:21,030 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:21,710 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:46:24,209 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:24,587 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:46:24,588 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:46:24,588 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:46:24,588 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:46:24,711 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:26,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:29,589 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:31,034 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:34,590 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:36,035 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:36,389 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:46:40,390 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:40,630 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:46:40,632 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:46:40,632 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:46:40,633 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:46:40,717 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:41,039 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:43,718 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:46:45,048 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:46:45,049 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:46:45,050 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:46:45,050 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:46:45,719 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:46:46,040 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:46,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:51,041 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:51,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:46:56,045 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:46:56,052 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:01,046 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:01,053 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:01,109 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:01,111 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:01,111 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:01,111 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:01,725 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:03,726 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:47:05,546 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:05,547 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:05,547 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:05,548 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:05,726 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:06,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:06,390 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:47:06,391 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:11,048 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:11,392 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:16,049 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:16,392 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:21,050 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:21,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:21,574 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:21,576 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:21,576 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:21,576 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:21,732 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:23,733 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:47:26,017 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:26,018 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:26,018 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:26,019 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:26,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:26,734 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:27,019 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:31,062 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:32,020 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:36,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:36,391 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:47:37,392 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:41,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:42,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:42,426 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:42,427 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:42,428 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:42,428 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:42,740 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:45,741 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:47:46,072 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:46,878 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:47:46,879 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:47:46,879 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:47:46,879 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:47:47,742 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:47:47,880 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:51,078 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:52,881 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:47:56,080 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:47:57,881 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:01,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:02,883 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:02,982 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:02,984 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:02,984 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:02,984 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:03,748 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:05,749 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:48:06,082 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:06,392 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:48:07,428 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:07,429 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:07,429 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:07,430 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:07,750 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:08,430 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:11,083 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:13,431 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:16,090 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:18,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:21,091 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:23,433 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:23,515 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:23,516 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:23,517 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:23,517 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:23,756 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:25,756 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:48:26,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:27,957 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:27,958 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:27,959 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:27,959 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:28,758 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:28,960 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:31,101 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:33,960 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:36,393 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:48:36,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:39,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:41,474 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:44,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:44,413 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:44,414 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:44,415 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:44,415 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:44,763 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:45,764 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:48:46,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:48,850 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:48:48,851 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:48:48,851 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:48:48,852 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:48:49,765 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:48:49,852 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:51,479 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:54,853 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:48:56,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:48:59,854 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:01,488 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:04,855 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:04,961 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:04,962 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:04,963 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:04,963 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:05,771 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:06,394 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:49:06,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:07,772 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:49:09,400 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:09,401 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:09,402 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:09,402 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:09,773 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:10,403 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:11,490 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:15,403 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:16,496 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:20,404 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:21,502 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:25,405 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:25,525 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:25,526 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:25,527 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:25,527 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:25,779 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:26,504 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:27,780 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:49:29,974 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:29,975 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:29,975 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:29,976 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:30,781 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:30,976 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:31,511 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:35,977 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:36,394 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:49:36,512 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:41,396 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:41,515 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:41,515 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:46,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:46,073 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:46,073 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:46,073 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:46,516 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:46,787 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:47,127 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:47,787 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:49:50,503 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:49:50,504 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:49:50,504 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:49:50,505 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:49:50,788 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:49:51,517 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:52,506 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:49:56,518 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:49:57,507 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:01,522 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:02,508 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:06,395 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:50:06,524 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:06,924 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:06,925 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:06,926 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:06,926 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:07,795 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:07,980 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:09,795 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:50:11,358 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:11,360 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:11,360 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:11,360 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:11,525 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:11,796 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:13,361 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:16,526 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:18,362 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:21,532 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:23,363 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:26,539 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:27,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:27,490 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:27,491 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:27,491 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:27,802 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:28,547 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:29,803 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:50:31,540 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:31,930 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:31,931 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:31,932 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:31,932 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:32,804 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:33,933 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:36,396 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:50:36,546 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:39,397 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:41,547 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:44,398 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:46,553 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:48,059 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:48,060 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:48,061 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:48,061 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:48,810 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:49,810 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:50:50,115 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:51,554 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:50:52,499 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:50:52,500 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:50:52,501 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:50:52,501 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:50:52,812 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:50:55,502 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:50:56,558 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:00,503 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:01,559 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:05,504 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:06,397 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:51:06,561 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:08,950 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:08,952 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:08,952 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:08,954 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:09,818 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:11,009 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:11,562 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:11,819 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:51:13,385 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:13,386 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:13,386 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:13,386 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:13,819 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:16,387 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:16,564 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:21,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:21,570 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:26,389 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:26,576 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:29,466 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:29,468 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:29,468 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:29,468 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:29,826 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:31,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:31,577 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:31,827 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:51:33,899 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:33,900 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:33,900 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:33,901 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:34,828 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:36,398 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:51:36,582 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:37,399 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:41,583 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:42,400 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:46,589 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:47,400 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:49,930 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:49,931 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:49,932 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:49,932 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:50,834 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:51,591 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:51,834 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:51:52,997 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:51:54,357 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:51:54,358 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:51:54,358 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:51:54,358 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:51:54,835 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:51:56,593 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:51:58,359 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:01,600 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:03,360 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:06,399 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:52:06,607 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:08,400 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:10,448 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:10,449 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:10,450 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:10,450 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:10,841 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:11,608 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:11,842 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:52:13,506 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:14,890 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:14,891 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:14,892 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:14,892 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:15,843 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:16,614 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:18,893 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:21,926 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:23,894 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:26,939 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:28,895 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:31,326 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:31,327 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:31,328 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:31,328 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:31,850 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:31,939 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:33,850 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:52:34,381 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:35,754 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:35,755 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:35,756 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:35,756 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:35,851 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:36,400 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:52:36,945 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:39,401 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:41,946 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:44,402 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:46,950 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:49,403 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:51,839 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:51,840 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:51,841 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:51,841 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:51,857 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:51,951 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:52:53,858 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:52:54,894 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:52:56,244 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:52:56,245 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:52:56,245 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:52:56,246 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:52:56,859 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:52:56,958 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:00,247 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:01,959 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:05,248 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:06,401 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:53:06,960 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:10,402 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:11,961 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:12,336 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:12,338 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:12,338 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:12,339 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:12,865 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:53:13,865 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:53:16,392 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:16,745 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:16,746 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:16,747 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:16,747 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:16,866 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:53:16,962 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:21,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:21,968 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:26,749 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:27,106 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:31,750 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:32,107 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:33,186 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:33,188 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:33,188 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:33,188 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:33,872 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:53:35,873 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:53:36,402 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:53:37,109 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:37,403 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:37,583 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:37,584 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:37,585 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:37,585 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:37,874 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:53:42,115 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:42,586 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:47,121 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:47,587 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:52,125 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:52,588 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:53,631 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:53,637 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:53,637 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:53,637 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:53,880 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:53:55,881 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:53:57,126 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:53:57,685 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:53:58,018 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:53:58,019 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:53:58,019 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:53:58,020 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:53:58,882 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:02,127 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:03,021 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:06,402 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:54:07,136 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:08,403 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:12,137 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:13,404 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:14,043 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:14,045 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:14,045 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:14,045 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:14,889 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:15,889 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:54:17,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:18,446 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:18,447 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:18,447 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:18,448 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:18,448 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:18,890 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:22,156 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:23,449 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:27,157 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:28,450 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:32,159 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:33,451 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:34,555 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:34,556 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:34,557 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:34,557 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:34,896 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:35,897 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:54:36,403 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:54:37,160 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:38,980 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:38,981 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:38,981 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:38,981 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:38,982 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:39,899 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:42,162 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:43,983 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:47,169 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:48,984 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:52,172 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:53,984 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:55,472 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:55,474 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:55,474 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:55,475 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:55,906 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:54:57,174 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:54:57,907 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:54:59,529 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:54:59,880 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:54:59,881 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:54:59,882 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:54:59,882 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:54:59,908 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:02,175 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:04,883 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:06,404 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:55:07,181 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:10,405 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:12,182 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:15,406 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:15,951 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:55:15,952 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:55:15,952 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:55:15,952 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:55:16,914 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:17,183 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:17,914 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:55:20,353 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:55:20,354 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:55:20,354 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:55:20,354 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:55:20,915 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:21,355 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:22,185 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:26,356 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:27,186 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:31,357 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:32,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:36,357 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:36,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:55:36,396 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:55:36,396 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:55:36,397 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:55:36,405 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:55:36,922 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:37,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:37,922 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:55:40,792 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:55:40,793 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:55:40,793 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:55:40,794 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:55:40,923 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:41,795 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:42,191 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:46,795 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:47,198 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:51,796 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:52,199 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:56,796 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:55:57,184 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:55:57,186 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:55:57,186 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:55:57,187 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:55:57,213 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:55:57,930 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:55:59,930 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:56:01,586 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:01,587 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:01,587 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:01,588 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:01,931 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:02,221 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:02,588 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:06,406 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:56:07,226 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:08,407 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:12,228 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:13,409 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:17,230 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:17,686 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:17,687 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:17,687 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:17,688 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:17,937 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:18,740 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:19,940 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:56:22,092 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:22,093 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:22,093 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:22,093 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:22,236 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:22,941 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:24,094 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:27,237 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:29,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:32,245 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:34,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:36,407 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:56:37,247 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:38,173 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:38,174 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:38,175 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:38,175 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:38,947 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:39,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:39,948 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:56:42,248 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:42,575 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:42,576 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:42,577 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:42,577 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:42,953 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:44,578 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:47,249 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:49,579 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:52,254 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:54,580 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:57,255 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:56:58,664 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:56:58,666 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:56:58,666 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:56:58,667 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:56:58,959 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:56:59,727 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:56:59,959 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:57:02,256 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:03,069 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:57:03,070 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:57:03,070 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:57:03,070 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:57:03,961 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:57:05,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:06,408 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:57:07,262 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:10,410 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:12,466 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:15,411 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:17,483 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:19,477 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:57:19,478 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:57:19,479 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:57:19,479 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:57:19,967 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:57:20,537 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:21,968 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:57:22,487 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:23,881 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:57:23,882 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:57:23,882 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:57:23,882 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:57:23,969 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:57:25,883 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:27,488 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:30,884 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:32,494 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:35,885 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:36,409 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:57:37,495 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:39,901 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:57:39,902 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:57:39,903 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:57:39,903 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:57:39,974 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:57:40,968 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:41,975 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:57:42,496 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:44,305 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:57:44,306 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:57:44,306 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:57:44,307 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:57:44,976 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:57:46,307 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:47,502 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:51,308 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:52,508 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:57:56,309 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:57:57,515 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:00,417 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:00,419 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:00,419 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:00,419 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:00,983 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:01,476 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:01,983 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:58:02,516 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:04,820 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:04,822 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:04,822 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:04,822 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:04,985 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:06,410 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:58:07,411 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:07,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:12,412 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:12,550 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:17,413 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:17,562 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:21,301 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:21,303 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:21,304 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:21,304 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:21,991 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:22,563 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:23,372 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:23,992 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:58:25,706 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:25,707 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:25,707 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:25,708 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:25,993 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:27,564 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:28,709 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:32,565 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:33,709 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:36,411 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:58:37,575 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:39,412 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:41,732 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:41,734 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:41,734 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:41,735 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:41,998 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:42,578 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:43,999 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:58:44,794 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:46,117 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:58:46,118 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:58:46,118 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:58:46,118 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:58:47,001 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:58:47,579 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:50,119 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:52,581 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:58:55,120 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:58:57,588 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:00,121 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:02,138 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:02,140 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:02,140 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:02,140 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:02,593 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:03,007 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:04,007 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:59:05,197 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:06,412 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:59:06,545 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:06,546 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:06,547 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:06,547 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:07,008 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:07,599 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:10,548 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:12,604 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:15,549 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:17,606 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:20,550 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:22,607 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:22,642 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:22,644 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:22,644 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:22,644 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:23,015 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:24,015 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:59:25,705 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:27,074 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:27,075 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:27,075 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:27,075 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:27,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:28,017 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:31,076 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:32,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:36,077 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:36,413 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 07:59:37,665 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:41,414 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:42,667 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:43,550 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:43,551 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:43,552 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:43,552 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:44,023 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:46,024 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 07:59:46,608 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:47,668 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:47,948 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 07:59:47,949 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 07:59:47,949 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 07:59:47,950 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 07:59:48,025 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 07:59:51,951 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:52,680 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 07:59:56,952 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 07:59:57,681 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:01,953 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:02,683 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:04,024 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:04,026 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:04,026 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:04,026 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:04,031 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:06,031 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:00:06,414 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:00:07,414 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:07,689 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:08,415 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:08,416 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:08,416 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:08,417 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:09,033 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:12,418 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:12,693 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:17,419 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:17,701 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:22,420 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:22,702 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:24,440 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:24,441 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:24,442 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:24,442 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:25,038 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:26,039 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:00:27,497 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:27,703 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:28,838 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:28,840 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:28,840 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:28,840 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:29,040 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:32,704 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:32,841 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:36,414 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:00:37,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:38,415 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:42,718 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:43,416 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:45,261 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:45,267 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:45,268 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:45,268 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:46,046 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:47,719 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:48,047 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:00:49,318 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:49,666 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:00:49,667 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:00:49,667 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:00:49,667 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:00:50,048 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:00:52,724 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:54,668 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:00:57,725 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:00:59,669 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:02,727 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:04,670 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:05,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:05,791 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:05,791 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:05,792 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:06,054 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:06,415 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:01:07,728 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:08,055 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:01:10,193 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:10,194 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:10,194 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:10,194 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:10,195 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:11,056 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:12,740 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:15,196 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:17,741 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:20,197 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:22,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:25,198 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:26,306 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:26,307 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:26,307 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:26,308 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:27,063 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:27,749 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:28,063 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:01:30,369 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:30,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:30,711 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:30,711 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:30,712 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:31,065 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:32,756 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:35,713 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:36,416 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:01:37,757 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:41,418 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:42,768 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:46,418 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:46,767 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:46,768 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:46,768 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:46,769 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:47,070 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:47,825 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:50,072 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:01:51,920 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:52,843 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:01:53,297 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:01:53,298 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:01:53,298 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:01:53,299 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:01:54,073 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:01:57,300 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:01:58,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:02,301 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:03,105 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:06,417 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:02:07,418 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:08,110 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:09,766 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:09,768 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:09,768 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:09,769 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:10,079 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:12,080 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:02:12,831 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:13,113 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:14,184 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:14,185 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:14,186 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:14,186 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:15,081 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:18,119 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:18,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:23,121 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:23,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:28,122 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:28,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:30,410 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:30,412 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:30,412 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:30,413 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:31,087 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:32,088 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:02:33,128 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:33,472 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:34,848 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:34,849 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:34,850 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:34,850 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:35,089 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:36,418 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:02:38,129 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:39,419 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:43,131 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:44,420 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:48,137 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:49,421 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:51,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:51,395 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:51,396 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:51,396 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:52,095 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:53,320 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:54,096 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:02:54,451 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:02:55,806 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:02:55,808 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:02:55,808 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:02:55,808 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:02:56,097 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:02:58,321 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:02:59,809 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:03,323 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:04,810 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:06,419 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:03:08,324 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:10,421 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:11,935 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:11,936 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:11,937 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:11,937 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:12,104 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:13,331 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:14,104 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:03:15,991 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:16,335 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:16,336 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:16,336 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:16,337 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:17,106 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:18,332 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:21,338 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:23,334 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:26,339 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:28,335 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:31,340 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:32,455 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:32,456 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:32,457 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:32,457 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:33,112 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:33,337 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:34,113 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:03:36,420 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:03:36,420 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:36,880 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:36,881 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:36,881 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:36,882 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:37,114 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:38,338 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:41,883 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:43,340 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:46,884 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:48,341 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:51,884 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:53,083 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:53,085 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:53,085 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:53,085 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:53,121 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:53,341 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:03:54,121 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:03:57,145 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:03:57,506 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:03:57,507 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:03:57,507 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:03:57,508 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:03:58,123 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:03:58,343 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:02,509 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:03,348 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:06,421 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:04:08,355 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:08,422 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:13,356 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:13,422 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:14,042 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:14,044 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:14,044 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:14,045 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:04:14,129 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:04:16,130 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:04:18,357 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:18,448 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:18,449 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:18,450 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:18,450 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:18,450 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:04:19,131 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:04:23,359 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:23,452 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:28,364 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:28,453 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:33,367 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:33,453 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:34,565 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:34,567 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:34,567 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:34,568 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:04:35,138 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:04:36,138 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:04:36,422 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:04:38,373 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:38,986 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:38,987 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:38,987 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:38,988 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:38,988 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:04:39,139 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:04:43,382 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:43,989 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:48,383 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:48,990 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:53,395 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:53,991 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:55,038 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:55,040 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:55,040 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:55,041 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:04:55,145 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:04:56,145 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:04:58,400 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:04:59,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:04:59,452 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:04:59,453 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:04:59,454 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:04:59,454 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:00,147 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:03,401 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:04,455 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:06,423 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:05:08,409 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:10,424 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:13,418 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:15,425 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:15,925 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:05:15,926 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:05:15,926 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:05:15,927 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:16,154 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:18,154 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:05:18,419 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:20,367 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:05:20,368 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:05:20,368 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:05:20,368 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:21,156 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:21,369 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:23,426 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:26,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:28,427 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:31,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:33,429 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:36,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:36,424 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:05:36,527 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:05:36,528 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:05:36,528 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:05:36,529 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:37,161 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:38,162 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:05:38,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:40,978 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:05:40,979 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:05:40,979 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:05:40,980 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:41,163 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:41,980 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:43,433 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:46,981 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:48,434 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:51,982 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:53,435 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:05:56,982 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:05:57,030 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:05:57,032 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:05:57,032 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:05:57,033 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:05:57,169 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:05:58,169 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:05:58,443 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:01,453 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:01,454 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:01,455 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:01,455 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:02,171 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:06:02,456 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:03,443 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:06,424 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:06:08,425 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:08,444 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:13,426 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:13,449 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:17,525 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:17,526 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:17,527 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:17,527 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:18,177 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:06:18,177 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:06:18,452 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:18,581 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:21,964 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:21,965 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:21,965 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:21,966 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:22,179 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:06:23,454 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:23,966 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:28,465 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:28,967 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:33,469 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:33,968 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:36,425 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:06:38,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:38,471 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:38,472 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:38,472 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:38,473 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:39,185 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:06:39,529 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:40,186 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:06:42,906 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:42,907 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:42,907 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:42,908 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:43,187 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:06:43,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:44,909 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:48,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:49,910 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:53,478 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:54,911 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:06:58,480 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:06:58,972 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:06:58,974 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:06:58,974 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:06:58,974 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:06:59,193 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:00,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:01,194 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:07:03,395 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:07:03,396 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:07:03,397 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:07:03,397 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:07:03,481 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:04,195 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:05,397 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:06,426 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:07:08,489 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:10,428 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:13,490 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:15,428 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:18,497 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:19,503 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:07:19,505 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:07:19,505 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:07:19,505 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:07:20,201 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:20,560 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:21,202 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:07:23,500 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:23,934 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:07:23,935 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:07:23,935 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:07:23,935 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:07:24,203 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:25,936 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:28,507 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:30,937 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:33,507 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:35,938 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:36,427 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:07:38,509 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:40,409 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:07:40,410 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:07:40,411 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:07:40,411 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:07:41,209 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:41,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:43,210 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:07:43,510 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:44,843 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:07:44,850 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:07:44,850 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:07:44,851 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:07:45,211 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:07:46,851 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:48,511 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:51,853 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:53,512 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:07:56,854 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:07:58,513 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:00,932 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:00,933 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:00,934 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:00,934 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:01,217 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:01,994 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:03,218 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:08:03,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:05,373 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:05,374 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:05,375 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:05,375 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:06,219 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:06,428 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:08:07,429 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:08,520 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:12,430 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:13,529 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:17,431 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:18,535 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:21,505 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:21,506 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:21,506 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:21,507 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:22,225 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:22,560 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:23,226 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:08:23,540 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:25,940 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:25,941 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:25,941 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:25,942 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:26,227 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:27,942 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:28,541 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:32,943 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:33,549 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:36,429 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:08:38,430 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:38,551 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:42,088 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:42,090 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:42,090 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:42,091 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:42,233 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:43,553 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:44,149 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:45,234 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:08:46,524 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:08:46,525 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:08:46,525 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:08:46,526 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:08:47,235 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:08:48,559 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:49,527 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:53,565 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:54,528 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:08:58,572 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:08:59,529 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:03,058 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:03,059 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:03,060 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:03,060 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:03,241 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:03,575 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:05,115 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:05,242 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:09:06,430 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:09:07,491 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:07,492 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:07,492 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:07,493 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:08,243 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:08,576 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:10,494 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:13,583 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:15,495 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:18,589 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:20,496 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:23,596 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:23,600 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:23,601 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:23,602 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:23,602 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:24,249 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:25,249 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:09:25,666 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:28,018 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:28,019 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:28,020 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:28,020 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:28,251 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:28,603 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:31,021 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:33,610 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:36,022 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:36,431 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:09:38,611 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:41,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:43,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:44,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:44,100 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:44,100 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:44,100 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:44,257 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:47,158 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:47,258 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:09:48,515 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:09:48,516 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:09:48,517 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:09:48,517 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:09:48,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:49,259 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:09:52,518 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:53,615 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:09:57,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:09:58,616 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:02,520 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:03,622 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:04,944 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:04,945 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:04,945 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:04,946 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:05,265 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:06,432 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:10:07,266 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:10:08,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:08,628 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:09,367 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:09,368 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:09,368 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:09,368 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:10,267 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:13,630 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:14,370 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:18,631 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:19,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:23,638 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:24,372 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:25,481 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:25,482 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:25,483 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:25,483 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:26,273 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:27,273 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:10:28,639 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:29,536 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:29,920 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:29,921 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:29,922 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:29,922 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:30,275 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:33,641 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:34,923 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:36,433 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:10:38,644 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:40,434 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:43,646 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:45,435 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:46,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:46,082 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:46,083 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:46,083 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:46,280 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:48,647 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:49,281 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:10:50,521 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:10:50,522 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:10:50,523 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:10:50,523 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:50,523 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:10:51,282 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:10:53,649 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:10:55,524 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:10:58,655 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:00,525 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:03,662 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:05,526 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:06,434 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:11:06,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:06,620 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:06,621 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:06,621 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:07,288 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:08,665 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:09,289 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:11:10,683 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:11,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:11,053 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:11,053 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:11,053 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:11,290 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:13,666 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:16,054 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:18,672 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:21,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:23,675 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:26,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:27,546 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:27,547 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:27,547 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:27,548 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:28,296 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:28,676 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:29,297 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:11:31,604 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:31,992 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:31,994 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:31,994 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:31,994 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:32,298 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:33,683 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:36,434 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:11:37,435 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:38,684 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:42,436 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:43,691 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:47,437 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:48,134 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:48,135 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:48,136 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:48,136 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:48,304 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:48,698 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:51,305 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:11:52,578 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:11:52,579 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:11:52,579 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:11:52,579 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:52,580 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:11:53,306 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:11:53,701 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:11:57,581 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:11:58,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:02,582 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:03,708 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:06,435 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:12:08,436 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:08,700 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:08,701 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:08,701 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:08,702 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:08,716 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:09,312 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:11,313 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:12:13,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:13,145 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:13,145 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:13,146 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:13,314 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:13,717 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:14,147 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:18,723 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:19,147 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:23,729 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:24,148 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:28,735 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:29,149 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:29,576 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:29,577 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:29,577 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:29,578 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:30,320 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:31,322 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:12:33,736 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:34,002 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:34,003 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:34,003 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:34,004 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:34,323 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:35,004 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:36,436 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:12:38,739 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:40,438 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:43,740 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:45,439 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:48,747 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:50,085 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:50,086 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:50,086 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:50,087 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:50,329 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:51,143 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:53,330 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:12:53,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:12:54,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:12:54,520 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:12:54,520 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:12:54,521 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:12:55,331 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:12:56,521 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:12:58,751 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:01,522 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:03,754 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:06,437 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:13:07,438 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:08,755 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:10,665 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:10,666 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:10,667 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:10,667 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:11,337 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:12,722 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:13,338 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:13:13,756 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:15,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:15,096 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:15,096 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:15,097 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:15,338 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:18,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:18,757 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:23,098 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:23,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:28,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:28,763 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:31,181 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:31,183 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:31,183 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:31,183 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:31,344 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:33,239 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:33,345 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:13:33,769 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:35,602 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:35,603 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:35,604 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:35,604 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:36,346 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:36,438 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:13:38,439 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:38,770 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:43,440 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:43,776 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:48,441 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:48,782 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:52,039 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:52,041 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:52,041 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:52,042 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:52,352 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:53,353 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:13:53,788 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:54,103 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:13:56,482 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:13:56,483 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:13:56,483 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:13:56,483 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:13:57,354 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:13:58,789 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:13:59,484 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:03,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:04,485 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:06,439 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:14:08,791 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:10,440 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:12,607 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:12,608 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:12,609 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:12,609 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:13,360 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:14:13,792 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:15,361 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:14:15,663 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:17,044 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:17,045 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:17,045 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:17,046 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:17,362 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:14:18,793 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:21,046 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:23,800 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:26,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:28,808 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:31,048 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:33,213 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:33,215 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:33,215 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:33,215 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:33,368 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:14:33,809 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:35,369 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:14:36,273 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:36,440 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:14:37,635 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:37,636 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:37,637 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:37,637 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:38,371 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:14:38,810 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:41,638 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:43,816 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:46,639 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:48,818 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:51,640 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:53,824 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:54,174 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:54,175 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:54,176 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:54,176 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:54,376 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:14:55,377 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:14:57,235 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:14:58,614 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:14:58,621 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:14:58,622 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:14:58,622 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:14:58,825 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:14:59,378 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:02,623 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:03,827 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:06,441 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:15:08,442 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:08,829 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:13,443 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:13,839 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:14,850 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:15:14,851 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:15:14,852 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:15:14,852 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:15:15,385 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:17,385 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:15:18,840 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:18,907 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:19,311 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:15:19,312 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:15:19,312 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:15:19,312 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:15:19,386 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:23,841 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:24,314 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:28,843 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:29,314 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:33,845 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:34,315 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:35,539 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:15:35,540 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:15:35,541 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:15:35,541 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:15:36,392 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:36,442 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:15:37,392 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:15:38,846 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:39,443 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:39,982 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:15:39,983 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:15:39,983 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:15:39,984 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:15:40,394 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:43,852 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:44,985 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:48,856 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:49,986 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:53,863 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:15:54,987 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:15:56,161 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:15:56,163 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:15:56,163 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:15:56,163 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:15:56,399 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:15:57,400 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:15:58,876 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:00,222 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:00,583 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:00,584 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:00,585 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:00,585 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:01,401 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:16:03,877 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:05,586 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:06,443 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:16:08,882 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:11,444 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:14,153 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:16,445 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:17,134 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:17,135 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:17,135 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:17,136 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:17,408 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:16:19,162 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:19,409 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:16:21,587 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:21,588 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:21,588 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:21,588 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:21,589 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:22,410 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:16:24,171 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:26,590 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:29,177 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:31,591 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:34,179 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:36,444 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:16:37,445 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:37,887 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:37,889 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:37,889 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:37,889 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:38,416 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:16:39,182 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:39,416 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:16:42,351 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:42,352 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:42,353 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:42,353 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:42,417 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:16:43,354 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:44,183 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:48,355 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:49,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:53,355 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:54,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:58,356 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:16:58,618 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:16:58,620 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:16:58,620 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:16:58,620 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:16:59,195 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:16:59,424 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:01,425 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:17:03,079 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:17:03,080 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:17:03,081 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:17:03,081 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:17:03,426 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:04,082 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:04,199 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:06,445 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:17:09,200 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:09,446 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:14,384 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:14,447 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:19,387 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:19,448 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:19,629 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:17:19,631 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:17:19,631 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:17:19,632 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:17:20,432 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:21,433 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:17:24,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:17:24,064 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:17:24,065 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:17:24,065 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:17:24,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:24,434 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:25,066 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:29,389 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:30,067 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:34,391 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:35,067 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:36,446 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:17:39,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:40,252 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:17:40,254 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:17:40,254 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:17:40,254 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:40,255 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:17:40,440 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:41,441 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:17:44,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:44,685 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:17:44,686 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:17:44,686 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:17:44,687 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:17:45,442 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:17:45,687 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:49,401 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:50,688 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:54,405 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:17:55,689 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:17:59,406 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:00,695 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:00,962 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:00,963 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:00,963 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:00,964 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:01,448 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:03,449 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:18:04,407 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:05,413 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:05,414 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:05,414 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:05,415 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:05,450 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:06,415 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:06,447 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:18:09,408 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:11,448 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:14,411 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:16,449 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:19,416 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:21,450 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:21,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:21,674 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:21,675 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:21,676 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:22,457 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:23,457 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:18:24,422 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:26,109 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:26,111 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:26,111 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:26,111 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:26,458 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:27,112 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:29,423 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:32,113 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:34,425 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:36,448 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:18:37,449 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:39,427 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:42,450 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:42,658 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:42,659 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:42,660 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:42,660 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:43,465 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:44,428 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:45,466 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:18:47,076 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:18:47,077 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:18:47,077 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:18:47,078 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:18:47,467 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:18:48,078 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:49,429 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:53,079 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:54,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:18:58,080 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:18:59,434 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:03,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:03,292 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:03,294 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:03,294 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:03,294 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:03,473 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:04,472 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:06,449 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:19:07,474 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:19:08,450 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:09,491 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:09,910 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:09,911 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:09,911 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:09,912 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:10,476 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:13,913 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:14,498 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:18,914 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:19,505 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:23,915 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:24,511 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:26,131 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:26,132 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:26,133 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:26,134 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:26,485 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:27,486 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:19:29,191 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:29,518 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:30,581 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:30,582 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:30,583 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:30,583 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:31,487 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:34,524 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:34,584 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:36,451 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:19:39,539 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:40,452 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:44,541 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:45,453 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:47,246 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:47,248 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:47,248 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:47,248 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:47,494 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:49,495 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:19:49,542 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:51,305 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:51,697 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:19:51,698 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:19:51,698 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:19:51,699 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:19:52,496 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:19:54,543 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:19:56,700 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:19:59,546 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:01,701 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:04,552 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:06,451 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:20:07,452 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:07,948 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:07,949 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:07,949 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:07,950 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:08,502 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:09,503 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:20:09,553 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:12,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:12,375 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:12,375 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:12,376 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:12,504 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:13,376 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:14,555 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:18,377 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:19,556 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:23,378 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:24,562 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:28,379 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:28,627 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:28,628 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:28,628 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:28,629 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:29,510 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:29,563 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:31,511 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:20:33,061 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:33,063 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:33,063 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:33,064 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:33,512 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:34,065 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:34,564 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:36,452 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:20:39,453 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:39,566 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:44,454 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:44,572 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:49,456 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:49,580 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:49,619 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:49,621 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:49,621 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:49,622 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:50,518 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:51,518 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:20:54,052 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:20:54,053 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:20:54,054 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:20:54,054 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:20:54,520 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:20:54,587 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:20:55,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:20:59,594 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:00,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:04,595 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:05,057 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:06,454 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:21:09,601 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:10,257 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:10,259 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:10,260 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:10,260 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:10,261 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:10,526 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:11,526 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:21:14,605 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:14,689 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:14,690 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:14,690 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:14,691 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:15,528 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:15,691 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:19,611 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:20,692 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:24,612 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:25,693 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:29,613 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:30,694 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:30,930 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:30,932 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:30,932 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:30,932 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:31,534 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:33,535 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:21:34,614 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:35,371 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:35,372 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:35,373 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:35,373 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:35,535 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:36,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:36,454 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:21:39,615 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:41,455 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:44,618 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:46,457 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:49,621 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:51,458 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:51,644 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:51,646 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:51,646 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:51,647 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:52,542 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:53,542 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:21:54,622 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:21:56,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:21:56,096 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:21:56,096 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:21:56,097 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:21:56,544 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:21:57,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:21:59,627 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:02,098 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:04,629 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:06,455 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:22:07,456 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:09,632 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:12,457 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:12,785 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:12,786 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:12,787 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:12,787 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:13,550 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:14,638 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:15,551 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:22:17,230 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:17,231 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:17,231 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:17,232 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:17,552 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:18,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:19,640 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:23,233 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:24,644 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:28,234 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:29,646 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:33,235 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:33,482 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:33,483 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:33,484 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:33,484 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:33,558 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:34,647 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:35,559 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:22:36,456 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:22:37,935 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:37,936 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:37,936 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:37,937 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:38,560 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:38,940 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:39,654 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:43,941 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:44,655 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:48,942 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:49,656 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:53,943 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:54,208 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:54,210 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:54,210 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:54,211 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:54,566 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:54,657 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:22:55,566 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:22:58,649 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:22:58,650 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:22:58,650 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:22:58,650 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:22:59,568 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:22:59,651 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:22:59,662 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:04,652 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:04,668 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:06,457 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:23:09,669 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:10,458 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:14,670 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:15,245 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:23:15,246 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:23:15,247 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:23:15,247 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:23:15,574 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:23:16,304 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:17,575 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:23:19,670 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:23:19,671 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:23:19,672 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:23:19,672 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:23:19,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:20,576 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:23:21,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:24,679 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:26,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:29,685 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:31,675 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:34,693 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:35,946 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:23:35,948 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:23:35,948 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:23:35,948 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:23:36,458 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:23:36,582 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:23:37,459 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:37,582 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:23:39,699 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:40,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:23:40,396 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:23:40,396 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:23:40,396 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:23:40,583 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:23:43,397 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:44,705 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:48,398 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:49,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:53,399 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:54,717 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:23:56,713 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:23:56,715 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:23:56,715 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:23:56,715 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:23:57,590 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:23:57,590 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:23:58,778 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:23:59,718 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:01,166 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:01,167 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:01,168 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:01,168 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:01,592 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:04,169 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:04,725 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:06,459 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:24:09,460 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:09,726 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:14,461 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:14,733 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:17,279 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:17,281 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:17,281 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:17,281 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:17,598 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:19,599 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:24:19,734 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:20,339 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:21,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:21,711 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:21,711 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:21,712 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:22,600 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:24,744 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:25,713 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:29,745 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:30,714 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:34,752 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:35,715 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:36,460 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:24:38,146 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:38,147 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:38,148 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:38,148 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:38,606 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:39,606 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:24:39,754 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:41,205 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:42,575 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:42,577 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:42,577 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:42,578 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:42,607 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:44,756 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:46,579 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:49,759 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:51,580 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:54,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:24:56,580 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:24:58,706 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:24:58,707 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:24:58,708 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:24:58,708 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:24:59,614 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:24:59,614 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:24:59,768 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:01,767 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:03,155 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:25:03,156 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:25:03,157 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:25:03,157 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:25:03,615 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:25:04,769 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:06,461 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:25:07,462 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:09,775 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:12,463 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:14,781 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:17,464 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:19,328 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:25:19,330 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:25:19,330 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:25:19,331 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:25:19,621 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:25:19,782 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:21,622 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:25:23,395 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:23,781 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:25:23,782 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:25:23,782 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:25:23,783 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:25:24,623 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:25:24,784 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:28,784 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:29,786 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:33,785 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:34,786 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:36,462 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:25:39,463 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:39,794 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:40,288 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:25:40,289 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:25:40,289 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:25:40,290 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:25:40,629 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:25:41,629 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:25:44,723 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:25:44,724 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:25:44,724 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:25:44,725 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:44,725 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:25:44,797 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:45,631 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:25:49,726 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:49,804 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:54,727 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:54,810 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:25:59,728 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:25:59,816 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:00,876 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:00,877 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:00,878 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:00,878 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:01,637 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:01,638 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:26:04,817 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:04,931 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:05,315 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:05,316 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:05,317 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:05,317 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:05,639 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:06,464 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:26:09,818 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:10,465 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:14,825 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:15,466 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:19,826 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:20,467 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:21,416 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:21,417 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:21,417 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:21,418 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:21,645 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:23,646 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:26:24,832 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:25,474 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:25,834 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:25,835 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:25,836 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:25,836 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:26,647 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:29,833 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:30,837 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:34,839 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:35,838 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:36,465 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:26:39,843 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:41,466 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:41,960 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:41,961 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:41,962 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:41,962 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:42,653 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:43,654 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:26:44,847 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:46,383 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:26:46,384 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:26:46,384 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:26:46,385 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:26:46,655 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:26:47,385 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:49,848 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:52,386 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:54,854 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:26:57,387 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:26:59,857 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:02,388 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:02,930 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:02,931 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:02,932 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:02,932 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:03,661 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:03,662 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:27:04,861 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:06,466 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:27:07,374 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:07,375 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:07,376 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:07,376 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:07,663 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:08,377 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:09,862 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:13,378 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:14,864 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:18,379 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:19,865 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:23,380 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:23,576 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:23,577 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:23,578 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:23,578 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:23,669 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:24,867 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:25,669 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:27:28,024 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:28,025 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:28,026 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:28,026 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:28,671 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:29,027 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:29,868 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:34,028 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:34,876 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:36,467 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:27:39,468 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:39,878 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:44,207 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:44,209 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:44,209 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:44,209 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:44,677 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:44,883 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:45,265 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:45,677 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:27:48,648 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:27:48,649 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:27:48,649 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:27:48,649 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:27:48,678 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:27:49,885 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:50,650 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:54,886 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:27:55,651 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:27:59,888 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:00,652 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:04,889 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:05,170 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:05,172 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:05,172 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:05,173 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:05,685 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:06,229 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:06,468 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:28:07,686 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:28:09,606 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:09,608 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:09,608 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:09,608 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:09,686 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:09,890 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:11,609 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:14,897 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:16,610 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:19,898 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:21,611 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:24,905 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:25,759 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:25,761 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:25,761 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:25,762 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:26,693 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:26,825 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:27,693 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:28:29,905 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:30,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:30,188 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:30,188 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:30,189 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:30,695 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:32,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:34,908 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:36,469 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:28:37,469 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:39,914 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:42,470 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:44,915 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:46,336 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:46,337 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:46,337 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:46,338 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:46,701 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:47,702 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:28:48,402 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:49,926 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:50,774 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:28:50,776 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:28:50,776 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:28:50,776 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:28:51,703 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:28:53,777 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:54,930 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:28:58,778 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:28:59,935 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:03,779 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:04,936 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:06,470 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:29:06,925 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:06,927 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:06,927 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:06,927 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:07,709 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:07,710 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:29:08,986 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:09,939 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:11,341 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:11,342 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:11,342 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:11,342 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:11,711 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:14,343 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:14,941 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:19,344 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:19,946 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:24,345 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:24,953 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:27,801 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:27,803 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:27,803 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:27,804 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:28,717 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:29,718 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:29:29,863 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:29,954 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:32,232 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:32,233 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:32,234 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:32,234 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:32,719 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:34,956 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:35,235 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:36,471 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:29:39,959 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:40,472 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:44,960 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:45,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:48,379 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:48,380 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:48,381 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:48,381 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:48,726 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:49,967 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:50,727 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:29:51,438 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:52,820 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:29:52,821 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:29:52,821 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:29:52,821 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:29:53,728 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:29:54,968 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:29:56,823 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:29:59,973 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:01,824 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:04,974 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:06,472 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:30:07,473 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:08,922 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:08,924 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:08,924 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:08,925 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:09,734 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:09,976 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:10,735 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:30:12,990 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:13,347 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:13,349 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:13,349 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:13,349 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:13,736 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:14,977 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:18,350 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:19,979 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:23,351 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:24,983 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:28,352 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:29,859 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:29,860 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:29,861 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:29,861 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:29,987 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:30,742 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:32,743 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:30:33,942 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:34,292 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:34,293 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:34,293 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:34,294 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:34,744 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:34,999 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:36,473 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:30:39,474 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:40,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:44,474 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:45,010 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:49,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:50,011 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:50,412 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:50,413 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:50,413 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:50,413 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:50,750 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:52,751 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:30:54,844 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:30:54,845 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:30:54,846 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:30:54,846 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:30:54,846 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:30:55,012 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:30:55,752 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:30:59,847 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:00,013 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:04,848 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:05,020 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:06,474 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:31:10,021 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:10,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:10,969 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:10,970 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:10,970 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:10,971 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:11,758 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:31:12,758 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:31:15,025 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:15,390 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:15,392 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:15,392 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:15,392 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:15,760 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:31:16,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:20,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:21,393 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:25,035 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:26,394 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:30,037 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:31,395 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:31,539 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:31,540 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:31,541 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:31,541 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:31,766 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:31:34,767 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:31:35,041 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:35,973 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:35,974 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:35,974 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:35,974 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:36,475 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:31:36,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:36,768 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:31:40,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:41,476 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:45,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:46,477 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:50,052 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:51,478 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:52,485 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:52,486 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:52,487 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:52,487 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:52,774 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:31:54,775 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:31:55,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:31:56,543 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:31:56,903 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:31:56,904 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:31:56,904 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:31:56,904 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:31:57,776 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:32:00,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:01,905 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:05,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:06,476 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:32:07,476 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:10,065 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:12,477 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:13,036 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:32:13,038 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:32:13,038 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:32:13,039 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:32:13,782 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:32:14,783 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:32:15,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:17,458 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:32:17,459 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:32:17,459 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:32:17,460 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:32:17,784 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:32:18,460 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:20,072 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:23,461 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:25,073 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:28,462 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:30,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:33,463 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:33,561 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:32:33,562 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:32:33,562 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:32:33,563 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:32:33,719 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: metric 2024-04-29 08:32:33,721 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:32:33,722 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:32:33,722 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:32:33,726 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:32:33,726 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:32:33,726 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:32:33,726 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:32:33,727 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:32:33,727 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:32:33,728 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:32:33,739 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:32:33,739 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:32:33,739 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:32:33,740 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:32:33,790 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:32:35,088 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:38,741 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:40,089 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:43,742 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:45,090 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:48,743 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:50,091 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:53,744 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:32:55,093 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:32:58,750 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:00,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:00,801 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/config.yaml 2024-04-29 08:33:05,095 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:05,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:10,096 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:10,100 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:15,097 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:15,101 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:20,099 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:20,101 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:23,265 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:33:23,265 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:33:23,265 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:33:23,265 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:33:23,266 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:33:23,266 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:33:23,268 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:33:23,268 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:33:23,269 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:33:24,810 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:33:25,100 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:25,329 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:30,113 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:30,330 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:35,124 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:35,331 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:40,137 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:40,332 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:45,142 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:45,333 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:50,143 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:50,334 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:33:52,777 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:33:52,778 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:33:52,778 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:33:52,781 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:33:52,821 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:33:54,822 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:33:55,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:33:55,836 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:00,145 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:00,837 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:05,154 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:05,838 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:08,359 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:34:08,360 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:34:08,361 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:34:08,361 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:34:08,827 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:34:10,160 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:10,828 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:34:11,420 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:15,170 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:16,421 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:18,212 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: partial_history 2024-04-29 08:34:18,213 DEBUG SenderThread:108 [sender.py:send():378] send: history 2024-04-29 08:34:18,214 DEBUG SenderThread:108 [sender.py:send_request():405] send_request: summary_record 2024-04-29 08:34:18,214 INFO SenderThread:108 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end 2024-04-29 08:34:18,224 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:34:18,224 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:34:18,224 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:34:18,225 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:34:18,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:34:18,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:34:18,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:34:18,227 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:34:18,239 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:34:18,239 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:34:18,240 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:34:18,240 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:34:18,831 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/wandb-summary.json 2024-04-29 08:34:20,172 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:22,241 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:25,173 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:27,242 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:30,174 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:32,243 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:35,175 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:37,244 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:40,177 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:42,245 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:43,697 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:34:43,697 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:34:43,697 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:34:43,697 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:34:43,698 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:34:43,699 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:34:43,700 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:34:43,700 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:34:43,700 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:34:43,701 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:34:43,701 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:34:43,701 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:34:43,701 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:34:43,702 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:34:43,704 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:34:43,704 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:34:43,704 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:34:45,178 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:47,706 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:50,179 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:52,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:34:55,180 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:34:57,708 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:00,181 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:02,709 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:05,182 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:07,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:10,183 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:12,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:15,184 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:17,712 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:20,185 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:22,713 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:25,186 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:27,714 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:30,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:32,715 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:35,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:37,716 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:40,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:42,717 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:45,191 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:47,718 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:50,192 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:51,216 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:35:51,216 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:35:51,216 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:35:51,217 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:35:51,217 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:35:51,218 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:35:51,219 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:35:51,220 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:35:51,220 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:35:51,224 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:35:51,224 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:35:51,224 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:35:51,224 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:35:51,225 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:35:51,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:35:51,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:35:51,225 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:35:51,243 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:35:51,243 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:35:51,243 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:35:51,243 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:35:53,244 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:35:55,193 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:35:55,650 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:35:55,650 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:35:55,650 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:35:55,651 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:35:55,652 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:35:55,652 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:35:55,653 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:35:55,653 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:35:55,654 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:35:58,172 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:35:58,172 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:35:58,172 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:35:58,173 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:35:58,173 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:35:58,173 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:35:58,174 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:35:58,174 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:35:58,185 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:35:58,185 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:35:58,185 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:35:58,186 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:35:59,186 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:00,287 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:04,187 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:05,289 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:09,188 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:10,290 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:14,189 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:15,291 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:19,190 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:20,292 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:23,490 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:36:23,491 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:36:23,491 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:36:23,491 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:36:23,492 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:36:23,492 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:36:23,493 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:36:23,494 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:36:23,495 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:36:23,500 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:36:23,500 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:36:23,500 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:36:23,501 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:36:23,501 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:36:23,501 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:36:23,501 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:36:23,501 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:36:23,517 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:36:23,518 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:36:23,518 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:36:23,518 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:36:24,475 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:36:24,475 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:36:24,475 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:36:24,475 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:36:24,476 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:36:24,476 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:36:24,477 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:36:24,478 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:36:24,479 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:36:24,519 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:25,299 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:25,329 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:36:25,329 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:36:25,329 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:36:25,330 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:36:25,330 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:36:25,330 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:36:25,330 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:36:25,331 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:36:25,343 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:36:25,343 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:36:25,343 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:36:25,344 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:36:30,305 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:30,345 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:35,306 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:35,346 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:40,307 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:40,347 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:45,308 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:45,348 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:47,421 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:36:47,421 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:36:47,421 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:36:47,421 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:36:47,422 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:36:47,422 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:36:47,423 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:36:47,424 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:36:47,425 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:36:47,427 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:36:47,427 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:36:47,427 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:36:47,427 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:36:47,427 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:36:47,428 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:36:47,430 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:36:47,431 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:36:47,449 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:36:47,449 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:36:47,449 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:36:47,449 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:36:48,364 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:36:48,364 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:36:48,364 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:36:48,364 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:36:48,365 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:36:48,366 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:36:48,368 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:36:48,369 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:36:48,370 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:36:48,682 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:36:48,682 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:36:48,682 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:36:48,683 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:36:48,683 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:36:48,684 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:36:48,683 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:36:48,684 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:36:48,695 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:36:48,695 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:36:48,695 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:36:48,696 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:36:50,309 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:50,696 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:36:55,311 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:36:55,697 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:00,312 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:00,698 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:05,313 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:05,700 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:10,314 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:10,701 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:15,316 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:15,702 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:20,317 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:20,703 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:25,318 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:25,704 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:26,761 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:37:26,762 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:37:26,762 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:37:26,762 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:37:26,762 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:37:26,764 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:37:26,765 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:37:26,766 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:37:26,767 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:37:27,659 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:37:27,659 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:37:27,659 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:37:27,659 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:37:27,660 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:37:27,660 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:37:27,661 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:37:27,660 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:37:27,671 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:37:27,671 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:37:27,671 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:37:27,672 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:37:30,641 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:31,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:31,817 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:37:31,817 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:37:31,817 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:37:31,817 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:37:31,818 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:37:31,818 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:37:31,819 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:37:31,820 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:37:31,820 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:37:31,850 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:37:31,850 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:37:31,850 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:37:31,850 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:37:31,851 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:37:31,851 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:37:31,851 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:37:31,852 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:37:31,862 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:37:31,862 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:37:31,862 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:37:31,863 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:37:35,642 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:36,864 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:40,643 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:41,865 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:45,644 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:46,867 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:48,264 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:37:48,264 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:37:48,264 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:37:48,264 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:37:48,265 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:37:48,265 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:37:48,266 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:37:48,267 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:37:48,268 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:37:48,998 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:37:48,998 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:37:48,998 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:37:48,999 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:37:48,999 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:37:48,999 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:37:48,999 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:37:48,999 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:37:49,011 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:37:49,011 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:37:49,011 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:37:49,012 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:37:50,649 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:52,013 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:37:55,650 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:37:57,014 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:00,651 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:02,015 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:05,652 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:07,016 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:08,126 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:38:08,127 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:38:08,127 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:38:08,127 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:38:08,127 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:38:08,128 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:38:08,131 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:38:08,131 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:38:08,132 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:38:09,035 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:38:09,035 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:38:09,035 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:38:09,035 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:38:09,036 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:38:09,036 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:38:09,036 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:38:09,036 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:38:09,047 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:38:09,047 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:38:09,047 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:38:09,048 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:38:10,653 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:12,014 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:38:12,015 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:38:12,015 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:38:12,015 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:38:12,016 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:38:12,016 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:38:12,017 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:38:12,018 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:38:12,020 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:38:12,041 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:38:12,041 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:38:12,041 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:38:12,041 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:38:12,042 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:38:12,042 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:38:12,043 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:38:12,043 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:38:12,053 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:38:12,054 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:38:12,054 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:38:12,054 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:12,054 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:38:15,655 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:17,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:20,656 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:22,057 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:24,976 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:38:24,976 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:38:24,976 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:38:24,976 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:38:24,977 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:38:24,977 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:38:24,979 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:38:24,980 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:38:24,981 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:38:25,121 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:38:25,121 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:38:25,121 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:38:25,122 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:38:25,122 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:38:25,122 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:38:25,122 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:38:25,122 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:38:25,133 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:38:25,133 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:38:25,133 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:38:25,134 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:38:25,657 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:27,135 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:30,658 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:32,136 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:35,659 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:37,137 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:40,660 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:42,138 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:45,661 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:47,139 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:50,663 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:52,140 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:38:55,664 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:38:57,141 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:00,665 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:02,142 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:05,666 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:07,143 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:10,667 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:12,144 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:15,668 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:17,146 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:20,669 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:22,146 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:25,670 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:27,147 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:30,671 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:32,148 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:35,672 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:37,149 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:40,673 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:42,150 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:43,661 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:39:43,661 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:39:43,661 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:39:43,661 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:39:43,662 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:39:43,664 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:39:43,665 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:39:43,666 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:39:43,666 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:39:43,985 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:39:43,986 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:39:43,986 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:39:43,986 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:39:43,986 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:39:43,987 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:39:43,987 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:39:43,988 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:39:44,000 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:39:44,001 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:39:44,001 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:39:44,001 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:39:45,675 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:48,002 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:50,676 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:53,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:39:55,677 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:39:57,432 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:39:57,432 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:39:57,432 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:39:57,432 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:39:57,433 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:39:57,434 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:39:57,435 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:39:57,435 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:39:57,436 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:39:57,729 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:39:57,729 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:39:57,729 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:39:57,729 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:39:57,729 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:39:57,729 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:39:57,730 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:39:57,730 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:39:57,742 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:39:57,742 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:39:57,742 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:39:57,743 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:39:58,743 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:00,497 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:00,497 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:00,497 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:00,498 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:00,498 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:00,499 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:00,500 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:00,501 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:00,501 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:00,678 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:00,705 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:00,705 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:00,705 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:00,706 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:00,706 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:00,706 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:00,707 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:00,707 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:00,718 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:00,718 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:00,718 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:00,718 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:04,199 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:04,199 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:04,199 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:04,199 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:04,200 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:04,201 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:04,202 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:04,202 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:04,205 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:04,257 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:04,257 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:04,257 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:04,257 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:04,257 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:04,258 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:04,258 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:04,258 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:04,269 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:04,269 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:04,269 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:04,270 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:04,270 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:05,679 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:07,105 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:07,105 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:07,105 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:07,105 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:07,106 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:07,107 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:07,107 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:07,108 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:07,109 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:07,186 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:07,186 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:07,186 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:07,187 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:07,187 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:07,187 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:07,192 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:07,192 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:07,198 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:07,199 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:07,199 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:07,199 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:10,200 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:10,680 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:13,434 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:13,434 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:13,434 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:13,434 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:13,435 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:13,435 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:13,436 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:13,437 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:13,440 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:13,508 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:13,508 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:13,508 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:13,508 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:13,508 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:13,509 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:13,509 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:13,509 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:13,521 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:13,521 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:13,521 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:13,521 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:15,522 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:15,681 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:17,190 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:17,190 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:17,191 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:17,191 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:17,191 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:17,192 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:17,193 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:17,197 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:17,197 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:17,289 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:17,289 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:17,289 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:17,290 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:17,290 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:17,290 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:17,290 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:17,290 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:17,303 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:17,303 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:17,303 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:17,307 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:20,682 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:21,308 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:23,263 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:23,263 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:23,263 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:23,263 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:23,264 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:23,264 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:23,266 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:23,267 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:23,276 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:23,276 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:23,276 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:23,276 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:23,276 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:23,276 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:23,276 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:23,277 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:23,278 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:23,299 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:23,299 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:23,299 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:23,299 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:25,683 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:25,969 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:27,300 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:30,684 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:32,301 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:33,905 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:33,905 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:33,905 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:33,905 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:33,906 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:33,906 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:33,907 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:33,908 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:33,911 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:35,686 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:35,974 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:36,981 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:36,981 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:36,981 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:36,981 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:36,981 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:36,981 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:36,984 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:36,984 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:36,994 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:36,995 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:36,995 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:36,995 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:37,975 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:37,996 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:39,975 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:40,687 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:42,997 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:43,897 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:40:43,897 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:40:43,897 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:40:43,897 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:40:43,898 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:40:43,899 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:40:43,899 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:40:43,900 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:40:43,903 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:40:45,688 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:45,978 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:47,029 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:40:47,029 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:40:47,029 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:40:47,029 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:40:47,029 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:40:47,030 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:40:47,030 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:40:47,031 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:40:47,043 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:40:47,043 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:40:47,043 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:40:47,043 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:40:47,979 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:48,044 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:49,980 INFO Thread-12 :108 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240429_072705-0t7z1bit/files/output.log 2024-04-29 08:40:50,689 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:53,045 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:40:55,690 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:40:58,046 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:00,691 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:03,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:05,692 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:08,048 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:10,694 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:13,049 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:15,695 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:18,050 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:20,695 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:23,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:25,696 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:28,052 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:30,697 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:33,054 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:35,698 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:38,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:40,699 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:43,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:45,700 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:48,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:50,701 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:53,057 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:41:55,702 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:41:58,058 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:00,703 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:03,059 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:05,705 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:08,060 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:10,706 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:13,061 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:15,707 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:18,062 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:20,708 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:23,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:25,709 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:28,064 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:30,710 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:33,065 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:35,711 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:38,066 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:40,712 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:43,067 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:45,712 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:48,068 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:50,714 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:53,069 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:42:55,715 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:42:58,070 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:00,716 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:03,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:05,717 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:08,072 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:10,719 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:13,073 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:15,720 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:18,074 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:20,721 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:23,075 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:25,722 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:28,076 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:30,723 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:33,077 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:35,724 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:38,078 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:40,725 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:43,078 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:45,726 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:48,079 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:50,728 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:53,080 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:43:55,729 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:43:58,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:00,730 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:03,082 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:05,731 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:08,083 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:10,732 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:13,084 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:15,734 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:18,085 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:20,735 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:23,086 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:25,736 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:28,086 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:30,737 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:33,087 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:35,738 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:38,088 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:40,739 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:43,089 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:45,740 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:48,089 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:50,741 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:53,090 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:44:55,742 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:44:58,091 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:00,743 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:03,093 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:05,744 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:08,094 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:10,745 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:12,811 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:45:12,811 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:45:12,811 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:45:12,811 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:45:12,812 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:45:12,823 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:45:12,824 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:45:12,836 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:45:12,836 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:45:12,981 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:45:12,981 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:45:12,982 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:45:12,982 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:45:12,982 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:45:12,982 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:45:12,983 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:45:12,983 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:45:12,999 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:45:12,999 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:45:12,999 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:45:13,000 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:45:14,001 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:15,747 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:19,001 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:20,748 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:24,002 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:25,749 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:29,003 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:30,750 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:34,005 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:35,751 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:39,006 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:40,752 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:44,007 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:45,753 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:49,008 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:50,754 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:54,009 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:45:55,755 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:45:59,010 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:00,756 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:04,011 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:05,758 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:09,012 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:10,759 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:14,014 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:15,760 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:19,014 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:20,761 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:24,015 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:25,762 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:29,016 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:30,763 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:34,017 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:35,764 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:39,018 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:40,766 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:44,019 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:45,767 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:49,020 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:50,768 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:54,021 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:46:55,769 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:46:59,022 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:00,770 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:04,023 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:05,771 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:09,024 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:10,772 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:14,025 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:15,773 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:19,026 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:20,774 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:24,028 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:25,776 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:29,029 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:30,777 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:34,030 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:35,778 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:39,031 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:40,779 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:44,032 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:45,780 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:49,033 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:50,781 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:54,034 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:47:55,782 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:47:59,034 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:00,783 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:04,036 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:05,784 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:09,037 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:10,785 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:14,038 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:15,787 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:19,039 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:20,788 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:24,040 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:25,789 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:29,041 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:30,790 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:34,042 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:35,792 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:39,043 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:40,793 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:44,044 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:45,794 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:49,045 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:50,795 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:54,046 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:48:55,796 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:48:59,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:00,797 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:04,047 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:05,798 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:09,048 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:10,799 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:14,049 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:15,800 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:19,050 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:20,801 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:24,051 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:25,802 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:29,053 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:30,803 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:34,054 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:35,804 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:39,055 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:40,805 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:44,056 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:45,806 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:49,057 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:50,807 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:54,058 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:49:55,809 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:49:59,059 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:00,810 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:04,060 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:05,810 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:09,061 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:10,811 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:14,062 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:15,812 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:19,063 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:20,814 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:24,064 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:25,815 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:29,065 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:30,816 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:34,066 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:35,817 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:39,067 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:40,818 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:44,069 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:45,820 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:49,070 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:50,821 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:54,071 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:50:55,822 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:50:59,072 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:00,823 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:04,073 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:05,824 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:09,074 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:10,825 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:14,075 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:15,826 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:19,076 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:20,827 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:24,077 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:25,828 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:29,078 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:30,829 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:34,079 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:35,830 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:39,080 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:40,831 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:44,081 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:45,832 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:49,082 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:50,834 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:54,083 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:51:55,835 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:51:59,084 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:00,836 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:04,085 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:05,837 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:09,086 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:10,838 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:14,087 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:15,839 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:19,088 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:20,840 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:24,089 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:25,841 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:29,090 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:30,842 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:31,675 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:52:31,675 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:52:31,676 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:52:31,676 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:52:31,676 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:52:31,677 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:52:31,678 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:52:31,681 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:52:31,681 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:52:34,091 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:35,860 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:39,092 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:40,861 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:44,093 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:45,861 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:49,094 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:50,249 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: pause 2024-04-29 08:52:50,250 INFO HandlerThread:108 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-29 08:52:50,250 INFO HandlerThread:108 [system_monitor.py:finish():203] Stopping system monitor 2024-04-29 08:52:50,250 INFO HandlerThread:108 [interfaces.py:finish():202] Joined cpu monitor 2024-04-29 08:52:50,251 INFO HandlerThread:108 [interfaces.py:finish():202] Joined disk monitor 2024-04-29 08:52:50,250 DEBUG SystemMonitor:108 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-29 08:52:50,251 DEBUG SystemMonitor:108 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-29 08:52:50,251 DEBUG SystemMonitor:108 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-29 08:52:50,262 INFO HandlerThread:108 [interfaces.py:finish():202] Joined gpu monitor 2024-04-29 08:52:50,262 INFO HandlerThread:108 [interfaces.py:finish():202] Joined memory monitor 2024-04-29 08:52:50,262 INFO HandlerThread:108 [interfaces.py:finish():202] Joined network monitor 2024-04-29 08:52:50,263 DEBUG SenderThread:108 [sender.py:send():378] send: stats 2024-04-29 08:52:50,862 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:54,264 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:52:55,863 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:52:59,265 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:53:00,865 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:53:04,266 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:53:05,866 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:53:09,267 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:53:10,867 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:53:14,267 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:53:15,868 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:53:19,268 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report 2024-04-29 08:53:20,869 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: keepalive 2024-04-29 08:53:21,913 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: resume 2024-04-29 08:53:21,913 INFO HandlerThread:108 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-29 08:53:21,913 INFO HandlerThread:108 [system_monitor.py:start():194] Starting system monitor 2024-04-29 08:53:21,913 INFO SystemMonitor:108 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-29 08:53:21,915 INFO SystemMonitor:108 [interfaces.py:start():190] Started cpu monitoring 2024-04-29 08:53:21,916 INFO SystemMonitor:108 [interfaces.py:start():190] Started disk monitoring 2024-04-29 08:53:21,918 INFO SystemMonitor:108 [interfaces.py:start():190] Started gpu monitoring 2024-04-29 08:53:21,918 INFO SystemMonitor:108 [interfaces.py:start():190] Started memory monitoring 2024-04-29 08:53:21,919 INFO SystemMonitor:108 [interfaces.py:start():190] Started network monitoring 2024-04-29 08:53:24,269 DEBUG HandlerThread:108 [handler.py:handle_request():146] handle_request: status_report