| 2024-04-08 18:47:16,822 INFO StreamThr :389 [internal.py:wandb_internal():86] W&B internal server running at pid: 389, started at: 2024-04-08 18:47:16.821419 | |
| 2024-04-08 18:47:16,823 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status | |
| 2024-04-08 18:47:17,195 INFO WriterThread:389 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/run-rbc2gze0.wandb | |
| 2024-04-08 18:47:17,195 DEBUG SenderThread:389 [sender.py:send():378] send: header | |
| 2024-04-08 18:47:17,198 DEBUG SenderThread:389 [sender.py:send():378] send: run | |
| 2024-04-08 18:47:17,414 INFO SenderThread:389 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files | |
| 2024-04-08 18:47:17,415 INFO SenderThread:389 [sender.py:_start_run_threads():1099] run started: rbc2gze0 with start time 1712602036.821012 | |
| 2024-04-08 18:47:17,422 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-04-08 18:47:17,423 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: check_version | |
| 2024-04-08 18:47:17,492 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-04-08 18:47:17,503 DEBUG HandlerThread:389 [system_info.py:__init__():26] System info init | |
| 2024-04-08 18:47:17,503 DEBUG HandlerThread:389 [system_info.py:__init__():41] System info init done | |
| 2024-04-08 18:47:17,503 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:47:17,504 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:47:17,504 INFO HandlerThread:389 [system_monitor.py:probe():214] Collecting system info | |
| 2024-04-08 18:47:17,504 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:47:17,505 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:47:17,506 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:47:17,507 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:47:17,508 INFO SystemMonitor:389 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 18:47:17,519 DEBUG HandlerThread:389 [system_info.py:probe():150] Probing system | |
| 2024-04-08 18:47:17,521 DEBUG HandlerThread:389 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-04-08 18:47:17,521 DEBUG HandlerThread:389 [system_info.py:probe():198] Probing system done | |
| 2024-04-08 18:47:17,521 DEBUG HandlerThread:389 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-08T18:47:17.519111', 'startedAt': '2024-04-08T18:47:16.815035', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '9d8ae58bd3dc', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.122, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.122, 'min': 0.0, 'max': 0.0}, {'current': 2000.122, 'min': 0.0, 'max': 0.0}, {'current': 2000.122, 'min': 0.0, 'max': 0.0}, {'current': 2000.122, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.724052429199}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-04-08 18:47:17,521 INFO HandlerThread:389 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-04-08 18:47:17,521 INFO HandlerThread:389 [system_monitor.py:probe():227] Publishing system info | |
| 2024-04-08 18:47:17,521 DEBUG HandlerThread:389 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-04-08 18:47:18,416 INFO Thread-12 :389 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/conda-environment.yaml | |
| 2024-04-08 18:48:12,432 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/conda-environment.yaml | |
| 2024-04-08 18:48:12,771 DEBUG HandlerThread:389 [system_info.py:_save_conda():219] Saving conda packages done | |
| 2024-04-08 18:48:12,772 INFO HandlerThread:389 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-04-08 18:48:12,777 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,777 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,777 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,777 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,778 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,779 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,780 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,780 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,780 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,780 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:12,780 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:12,781 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: python_packages | |
| 2024-04-08 18:48:12,784 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: stop_status | |
| 2024-04-08 18:48:12,902 DEBUG SenderThread:389 [sender.py:send():378] send: telemetry | |
| 2024-04-08 18:48:12,902 DEBUG SenderThread:389 [sender.py:send():378] send: config | |
| 2024-04-08 18:48:12,904 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:12,904 DEBUG SenderThread:389 [sender.py:send():378] send: telemetry | |
| 2024-04-08 18:48:12,905 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:12,905 WARNING SenderThread:389 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-04-08 18:48:12,905 DEBUG SenderThread:389 [sender.py:send():378] send: telemetry | |
| 2024-04-08 18:48:12,906 DEBUG SenderThread:389 [sender.py:send():378] send: files | |
| 2024-04-08 18:48:12,906 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-metadata.json with policy now | |
| 2024-04-08 18:48:13,198 INFO wandb-upload_0:389 [upload_job.py:push():131] Uploaded file /tmp/tmphx45kdr4wandb/80stm69h-wandb-metadata.json | |
| 2024-04-08 18:48:13,432 INFO Thread-12 :389 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-metadata.json | |
| 2024-04-08 18:48:13,433 INFO Thread-12 :389 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/requirements.txt | |
| 2024-04-08 18:48:13,433 INFO Thread-12 :389 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:48:14,471 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:15,433 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:48:17,508 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:48:17,509 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:48:17,515 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:18,434 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/config.yaml | |
| 2024-04-08 18:48:19,472 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:22,713 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:23,292 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:48:23,294 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:23,294 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:23,295 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:23,295 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:23,295 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:48:23,295 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:48:23,295 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:48:23,436 INFO Thread-12 :389 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:48:24,472 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:25,437 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:48:28,197 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:29,508 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:33,198 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:34,515 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:38,199 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:39,521 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:43,200 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:44,527 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:47,510 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:48:48,516 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:49,447 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/config.yaml | |
| 2024-04-08 18:48:49,528 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:53,374 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:48:53,376 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:48:53,376 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:48:53,377 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:48:53,448 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:48:53,853 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:48:53,855 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:48:53,856 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:48:53,857 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:53,857 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:48:54,449 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:48:54,530 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:48:55,449 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:48:55,871 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 18:48:55,871 INFO HandlerThread:389 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 18:48:55,871 INFO HandlerThread:389 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 18:48:55,871 DEBUG SystemMonitor:389 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 18:48:55,871 DEBUG SystemMonitor:389 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 18:48:55,872 INFO HandlerThread:389 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 18:48:55,873 INFO HandlerThread:389 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 18:48:55,885 INFO HandlerThread:389 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 18:48:55,885 INFO HandlerThread:389 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 18:48:55,885 INFO HandlerThread:389 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-08 18:48:55,886 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:48:58,887 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:48:59,536 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:03,888 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:04,537 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:08,889 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:09,538 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:13,890 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:14,539 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:18,896 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:19,457 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/config.yaml | |
| 2024-04-08 18:49:19,540 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:24,021 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:24,541 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:29,021 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:29,542 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:34,022 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:34,543 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:39,023 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:39,544 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:44,025 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:44,545 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:49,026 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:49,546 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:50,241 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: resume | |
| 2024-04-08 18:49:50,241 INFO HandlerThread:389 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-04-08 18:49:50,241 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:49:50,241 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:49:50,242 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:49:50,243 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:49:50,243 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 18:49:50,244 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:49:50,244 INFO HandlerThread:389 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 18:49:50,245 INFO HandlerThread:389 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 18:49:50,245 INFO HandlerThread:389 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 18:49:50,246 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:49:50,246 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:49:50,246 DEBUG SystemMonitor:389 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 18:49:50,246 DEBUG SystemMonitor:389 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 18:49:50,248 INFO HandlerThread:389 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 18:49:50,268 INFO HandlerThread:389 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 18:49:50,268 INFO HandlerThread:389 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 18:49:50,268 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:49:54,269 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:54,547 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:49:59,270 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:49:59,548 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:04,272 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:04,549 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:09,272 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:09,550 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:14,274 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:14,551 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:19,275 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:19,552 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:22,045 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: resume | |
| 2024-04-08 18:50:22,046 INFO HandlerThread:389 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-04-08 18:50:22,046 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:50:22,046 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:50:22,046 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:50:22,047 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:50:22,048 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:50:22,048 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:50:22,049 INFO SystemMonitor:389 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 18:50:24,276 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:24,593 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:26,442 DEBUG SenderThread:389 [sender.py:send():378] send: config | |
| 2024-04-08 18:50:26,444 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:50:26,444 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:50:26,444 WARNING SenderThread:389 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-04-08 18:50:27,482 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:50:29,483 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:50:29,594 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:30,029 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:34,595 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:35,030 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:39,596 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:40,031 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:44,603 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:45,031 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:49,731 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:50,038 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:50,491 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/config.yaml | |
| 2024-04-08 18:50:54,737 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:50:55,182 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:50:59,392 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:50:59,393 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:50:59,393 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:50:59,395 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:50:59,494 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:50:59,738 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:00,376 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:01,495 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:51:04,739 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:05,377 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:10,001 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:10,378 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:15,010 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:15,379 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:20,012 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:20,380 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:22,049 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:51:22,051 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:51:25,013 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:26,052 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:28,650 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:51:28,652 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:51:28,652 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:51:28,653 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:51:29,272 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:51:29,273 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:51:29,274 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:51:29,274 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:51:29,436 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 18:51:29,436 INFO HandlerThread:389 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 18:51:29,436 INFO HandlerThread:389 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 18:51:29,437 DEBUG SystemMonitor:389 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 18:51:29,437 DEBUG SystemMonitor:389 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 18:51:29,438 INFO HandlerThread:389 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 18:51:29,438 INFO HandlerThread:389 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 18:51:29,452 INFO HandlerThread:389 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 18:51:29,452 INFO HandlerThread:389 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 18:51:29,452 INFO HandlerThread:389 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-08 18:51:29,453 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:51:29,506 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:51:29,506 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:51:30,014 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:31,453 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:35,015 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:36,454 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:40,016 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:41,455 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:45,017 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:46,457 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:50,018 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:51,458 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:51:55,020 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:51:56,459 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:00,021 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:01,460 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:05,022 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:06,461 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:10,023 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:11,462 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:15,024 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:16,463 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:20,025 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:21,464 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:25,026 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:26,465 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:30,027 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:31,466 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:35,028 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:36,467 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:40,029 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:41,468 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:45,030 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:46,469 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:50,031 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:51,470 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:52:55,033 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:52:56,471 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:00,034 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:01,472 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:05,035 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:06,473 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:10,036 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:11,474 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:15,037 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:16,475 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:20,038 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:21,476 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:25,039 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:26,477 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:30,040 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:31,478 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:35,041 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:36,479 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:40,042 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:41,480 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:45,043 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:46,481 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:50,044 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:51,482 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:53:55,045 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:53:56,483 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:00,047 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:01,484 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:05,048 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:06,486 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:10,049 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:11,487 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:15,050 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:16,488 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:17,136 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: resume | |
| 2024-04-08 18:54:17,136 INFO HandlerThread:389 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-04-08 18:54:17,136 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:54:17,137 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:54:17,137 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:54:17,138 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:54:17,139 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:54:17,140 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:54:17,140 INFO SystemMonitor:389 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 18:54:17,160 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 18:54:17,160 INFO HandlerThread:389 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 18:54:17,160 INFO HandlerThread:389 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 18:54:17,160 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:54:17,160 DEBUG SystemMonitor:389 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 18:54:17,161 DEBUG SystemMonitor:389 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 18:54:17,160 INFO HandlerThread:389 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 18:54:17,162 INFO HandlerThread:389 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 18:54:17,176 INFO HandlerThread:389 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 18:54:17,176 INFO HandlerThread:389 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 18:54:17,176 INFO HandlerThread:389 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-08 18:54:17,177 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:54:20,051 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:22,178 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:25,052 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:27,179 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:30,053 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:32,180 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:35,054 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:37,181 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:40,055 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:42,182 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:45,056 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:47,183 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:50,058 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:52,185 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:54:55,059 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:54:57,186 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:00,060 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:02,186 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:05,061 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:07,187 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:10,062 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:12,188 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:15,063 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:17,189 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:20,064 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:22,190 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:25,066 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:27,191 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:30,067 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:32,192 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:35,068 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:37,194 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:40,069 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:42,195 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:45,070 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:47,195 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:50,071 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:52,197 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:55:55,072 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:55:55,846 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: resume | |
| 2024-04-08 18:55:55,846 INFO HandlerThread:389 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-04-08 18:55:55,846 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:55:55,846 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:55:55,847 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:55:55,848 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:55:55,848 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:55:55,848 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: pause | |
| 2024-04-08 18:55:55,849 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:55:55,849 INFO HandlerThread:389 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-04-08 18:55:55,850 INFO HandlerThread:389 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-04-08 18:55:55,850 INFO HandlerThread:389 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-04-08 18:55:55,851 INFO SystemMonitor:389 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 18:55:55,851 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:55:55,851 DEBUG SystemMonitor:389 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-04-08 18:55:55,851 DEBUG SystemMonitor:389 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-04-08 18:55:55,854 INFO HandlerThread:389 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-04-08 18:55:55,872 INFO HandlerThread:389 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-04-08 18:55:55,872 INFO HandlerThread:389 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-04-08 18:55:55,872 INFO HandlerThread:389 [interfaces.py:finish():202] Joined network monitor | |
| 2024-04-08 18:55:55,873 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:55:57,873 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:00,073 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:02,874 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:05,074 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:07,876 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:08,163 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: resume | |
| 2024-04-08 18:56:08,163 INFO HandlerThread:389 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-04-08 18:56:08,163 INFO HandlerThread:389 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-08 18:56:08,163 INFO SystemMonitor:389 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-08 18:56:08,164 INFO SystemMonitor:389 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-08 18:56:08,165 INFO SystemMonitor:389 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-08 18:56:08,167 INFO SystemMonitor:389 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-08 18:56:08,167 INFO SystemMonitor:389 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-08 18:56:08,168 INFO SystemMonitor:389 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-08 18:56:10,076 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:12,353 DEBUG SenderThread:389 [sender.py:send():378] send: config | |
| 2024-04-08 18:56:12,355 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:56:12,355 DEBUG SenderThread:389 [sender.py:send():378] send: metric | |
| 2024-04-08 18:56:12,355 WARNING SenderThread:389 [sender.py:send_metric():1316] Seen metric with glob (shouldn't happen) | |
| 2024-04-08 18:56:12,942 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:13,614 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:56:15,288 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:17,943 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:20,288 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:22,949 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:23,617 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/config.yaml | |
| 2024-04-08 18:56:25,295 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:28,138 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:30,301 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:33,139 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:35,307 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:38,140 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:40,309 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:43,141 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:45,315 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:45,509 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:56:45,510 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:56:45,511 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:56:45,511 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:56:45,625 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:56:47,626 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |
| 2024-04-08 18:56:48,415 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:50,337 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:53,416 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:56:55,350 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:56:58,417 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:57:00,351 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:57:03,418 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:57:05,352 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:57:08,168 DEBUG SystemMonitor:389 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-08 18:57:08,169 DEBUG SenderThread:389 [sender.py:send():378] send: stats | |
| 2024-04-08 18:57:09,170 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:57:10,354 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:57:14,171 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-08 18:57:14,592 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:57:14,593 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:57:14,594 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:57:14,596 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:57:14,636 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:57:15,084 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-08 18:57:15,086 DEBUG SenderThread:389 [sender.py:send():378] send: history | |
| 2024-04-08 18:57:15,086 DEBUG SenderThread:389 [sender.py:send_request():405] send_request: summary_record | |
| 2024-04-08 18:57:15,086 INFO SenderThread:389 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end | |
| 2024-04-08 18:57:15,355 DEBUG HandlerThread:389 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-04-08 18:57:15,636 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/wandb-summary.json | |
| 2024-04-08 18:57:15,637 INFO Thread-12 :389 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240408_184716-rbc2gze0/files/output.log | |