2024-04-11 11:59:58,693 INFO StreamThr :472 [internal.py:wandb_internal():86] W&B internal server running at pid: 472, started at: 2024-04-11 11:59:58.693009 2024-04-11 11:59:58,697 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status 2024-04-11 11:59:59,019 INFO WriterThread:472 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/run-sm65qpc7.wandb 2024-04-11 11:59:59,020 DEBUG SenderThread:472 [sender.py:send():379] send: header 2024-04-11 11:59:59,022 DEBUG SenderThread:472 [sender.py:send():379] send: run 2024-04-11 11:59:59,164 INFO SenderThread:472 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files 2024-04-11 11:59:59,164 INFO SenderThread:472 [sender.py:_start_run_threads():1124] run started: sm65qpc7 with start time 1712836798.693162 2024-04-11 11:59:59,173 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: check_version 2024-04-11 11:59:59,174 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: check_version 2024-04-11 11:59:59,266 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: run_start 2024-04-11 11:59:59,277 DEBUG HandlerThread:472 [system_info.py:__init__():26] System info init 2024-04-11 11:59:59,277 DEBUG HandlerThread:472 [system_info.py:__init__():41] System info init done 2024-04-11 11:59:59,277 INFO HandlerThread:472 [system_monitor.py:start():194] Starting system monitor 2024-04-11 11:59:59,277 INFO SystemMonitor:472 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-11 11:59:59,278 INFO SystemMonitor:472 [interfaces.py:start():190] Started cpu monitoring 2024-04-11 11:59:59,278 INFO HandlerThread:472 [system_monitor.py:probe():214] Collecting system info 2024-04-11 11:59:59,279 INFO SystemMonitor:472 [interfaces.py:start():190] Started disk monitoring 2024-04-11 11:59:59,280 INFO SystemMonitor:472 [interfaces.py:start():190] Started gpu monitoring 2024-04-11 11:59:59,281 INFO SystemMonitor:472 [interfaces.py:start():190] Started memory monitoring 2024-04-11 11:59:59,282 INFO SystemMonitor:472 [interfaces.py:start():190] Started network monitoring 2024-04-11 11:59:59,292 DEBUG HandlerThread:472 [system_info.py:probe():150] Probing system 2024-04-11 11:59:59,295 DEBUG HandlerThread:472 [gitlib.py:_init_repo():56] git repository is invalid 2024-04-11 11:59:59,295 DEBUG HandlerThread:472 [system_info.py:probe():198] Probing system done 2024-04-11 11:59:59,295 DEBUG HandlerThread:472 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-11T11:59:59.292356', 'startedAt': '2024-04-11T11:59:58.685383', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '55cd788bb41a', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.18, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.18, 'min': 0.0, 'max': 0.0}, {'current': 2000.18, 'min': 0.0, 'max': 0.0}, {'current': 2000.18, 'min': 0.0, 'max': 0.0}, {'current': 2000.18, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5568.725933074951}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} 2024-04-11 11:59:59,295 INFO HandlerThread:472 [system_monitor.py:probe():224] Finished collecting system info 2024-04-11 11:59:59,295 INFO HandlerThread:472 [system_monitor.py:probe():227] Publishing system info 2024-04-11 11:59:59,295 DEBUG HandlerThread:472 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-04-11 12:00:00,167 INFO Thread-12 :472 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/conda-environment.yaml 2024-04-11 12:00:14,311 ERROR HandlerThread:472 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-04-11 12:00:14,312 DEBUG HandlerThread:472 [system_info.py:_save_conda():222] Saving conda packages done 2024-04-11 12:00:14,312 INFO HandlerThread:472 [system_monitor.py:probe():229] Finished publishing system info 2024-04-11 12:00:14,317 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:14,317 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: keepalive 2024-04-11 12:00:14,317 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:14,317 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: keepalive 2024-04-11 12:00:14,318 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:14,318 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: keepalive 2024-04-11 12:00:14,318 DEBUG SenderThread:472 [sender.py:send():379] send: files 2024-04-11 12:00:14,318 INFO SenderThread:472 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now 2024-04-11 12:00:14,510 INFO wandb-upload_0:472 [upload_job.py:push():131] Uploaded file /tmp/tmpduacgb07wandb/hbppccgg-wandb-metadata.json 2024-04-11 12:00:15,169 INFO Thread-12 :472 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-metadata.json 2024-04-11 12:00:15,245 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: python_packages 2024-04-11 12:00:15,245 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: python_packages 2024-04-11 12:00:15,248 DEBUG SenderThread:472 [sender.py:send():379] send: telemetry 2024-04-11 12:00:15,258 DEBUG SenderThread:472 [sender.py:send():379] send: config 2024-04-11 12:00:15,260 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:00:15,260 DEBUG SenderThread:472 [sender.py:send():379] send: telemetry 2024-04-11 12:00:15,260 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:00:15,261 WARNING SenderThread:472 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-04-11 12:00:15,261 DEBUG SenderThread:472 [sender.py:send():379] send: telemetry 2024-04-11 12:00:15,262 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:00:15,262 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:00:15,266 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:00:16,170 INFO Thread-12 :472 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:16,170 INFO Thread-12 :472 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/requirements.txt 2024-04-11 12:00:18,171 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:19,818 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:20,171 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:22,172 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:24,819 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:29,825 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:30,176 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/config.yaml 2024-04-11 12:00:30,247 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:00:30,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:00:30,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:00:35,316 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:38,179 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:40,779 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:45,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:00:45,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:00:45,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:00:46,376 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:51,377 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:57,292 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:00:58,187 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:00:59,282 DEBUG SystemMonitor:472 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-11 12:00:59,284 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:01:00,246 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:01:00,247 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:01:00,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:01:02,314 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:07,315 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:12,403 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:14,193 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:01:15,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:01:15,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:01:15,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:01:18,373 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:23,374 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:24,481 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: pause 2024-04-11 12:01:24,481 INFO HandlerThread:472 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-11 12:01:24,481 INFO HandlerThread:472 [system_monitor.py:finish():203] Stopping system monitor 2024-04-11 12:01:24,481 DEBUG SystemMonitor:472 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-11 12:01:24,482 DEBUG SystemMonitor:472 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-11 12:01:24,482 INFO HandlerThread:472 [interfaces.py:finish():202] Joined cpu monitor 2024-04-11 12:01:24,483 INFO HandlerThread:472 [interfaces.py:finish():202] Joined disk monitor 2024-04-11 12:01:24,494 INFO HandlerThread:472 [interfaces.py:finish():202] Joined gpu monitor 2024-04-11 12:01:24,494 INFO HandlerThread:472 [interfaces.py:finish():202] Joined memory monitor 2024-04-11 12:01:24,494 INFO HandlerThread:472 [interfaces.py:finish():202] Joined network monitor 2024-04-11 12:01:24,494 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:01:28,495 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:30,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:01:30,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:01:30,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:01:34,384 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:39,385 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:44,386 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:45,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:01:45,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:01:45,286 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:01:50,287 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:01:55,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:00,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:02:00,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:02:00,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:02:00,374 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:05,375 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:10,376 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:15,248 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:02:15,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:02:15,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:02:16,306 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:21,307 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:26,307 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:30,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:02:30,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:02:30,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:02:31,328 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:36,329 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:41,330 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:45,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:02:45,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:02:45,290 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:02:46,368 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:51,370 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:02:56,370 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:00,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:03:00,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:03:00,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:03:02,305 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:07,305 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:12,306 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:15,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:03:15,249 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:03:15,290 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:03:17,314 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:22,315 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:27,317 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:30,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:03:30,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:03:30,290 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:03:32,344 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:37,345 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:42,345 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:45,249 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:03:45,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:03:45,290 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:03:48,331 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:53,332 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:03:58,333 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:00,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:04:00,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:04:00,290 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:04:04,304 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:09,305 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:14,306 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:15,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:04:15,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:04:15,291 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:04:20,293 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:25,294 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:30,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:04:30,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:04:30,291 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:04:30,318 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:35,320 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:40,320 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:45,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:04:45,250 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:04:45,291 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:04:45,793 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: resume 2024-04-11 12:04:45,793 INFO HandlerThread:472 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-11 12:04:45,793 INFO HandlerThread:472 [system_monitor.py:start():194] Starting system monitor 2024-04-11 12:04:45,793 INFO SystemMonitor:472 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-11 12:04:45,794 INFO SystemMonitor:472 [interfaces.py:start():190] Started cpu monitoring 2024-04-11 12:04:45,794 INFO SystemMonitor:472 [interfaces.py:start():190] Started disk monitoring 2024-04-11 12:04:45,795 INFO SystemMonitor:472 [interfaces.py:start():190] Started gpu monitoring 2024-04-11 12:04:45,795 INFO SystemMonitor:472 [interfaces.py:start():190] Started memory monitoring 2024-04-11 12:04:45,797 INFO SystemMonitor:472 [interfaces.py:start():190] Started network monitoring 2024-04-11 12:04:46,308 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:51,309 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:04:56,310 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:00,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:05:00,251 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:05:00,253 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:05:01,348 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:06,349 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:11,349 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:15,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:05:15,251 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:05:15,291 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:05:17,338 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:22,339 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:27,340 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:30,250 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:05:30,251 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:05:30,291 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:05:32,357 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:37,358 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:42,359 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:45,251 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:05:45,251 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:05:45,292 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:05:45,797 DEBUG SystemMonitor:472 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-11 12:05:45,798 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:05:47,799 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:52,800 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:05:57,801 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:00,251 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:06:00,251 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:06:00,274 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:06:03,275 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:08,276 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:13,277 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:15,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:06:15,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:06:15,257 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:06:15,801 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:06:18,802 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:23,803 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:28,804 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:30,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:06:30,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:06:30,257 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:06:34,358 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:39,359 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:44,360 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:45,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:06:45,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:06:45,257 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:06:45,802 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:06:49,803 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:54,804 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:06:59,805 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:00,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:07:00,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:07:00,257 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:07:05,347 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:10,348 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:15,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:07:15,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:07:15,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:07:15,804 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:07:15,806 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:20,805 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:25,806 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:30,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:07:30,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:07:30,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:07:31,382 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:36,383 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:41,384 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:45,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:07:45,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:07:45,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:07:45,804 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:07:46,805 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:51,806 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:07:56,807 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:00,252 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:08:00,257 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:08:00,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:08:02,384 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:07,471 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:07,549 DEBUG SenderThread:472 [sender.py:send():379] send: config 2024-04-11 12:08:07,552 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:08:07,552 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:08:07,552 WARNING SenderThread:472 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-04-11 12:08:08,360 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:08:10,361 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:08:12,915 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:15,253 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:08:15,258 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:08:15,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:08:15,805 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:08:18,812 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:19,364 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/config.yaml 2024-04-11 12:08:23,957 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:26,367 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:08:29,061 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:30,253 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:08:30,258 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:08:30,258 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:08:34,293 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:40,133 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:40,372 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:08:45,134 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:45,254 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:08:45,258 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:08:45,259 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:08:45,807 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:08:50,808 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:08:55,809 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:00,254 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:09:00,260 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:09:00,261 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:09:01,307 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:06,307 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:07,071 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: partial_history 2024-04-11 12:09:07,073 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:09:07,074 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:09:07,074 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:09:07,074 DEBUG SenderThread:472 [sender.py:send():379] send: metric 2024-04-11 12:09:07,074 DEBUG SenderThread:472 [sender.py:send():379] send: history 2024-04-11 12:09:07,074 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: summary_record 2024-04-11 12:09:07,074 INFO SenderThread:472 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-11 12:09:07,383 INFO Thread-12 :472 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-summary.json 2024-04-11 12:09:08,383 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:11,449 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:12,385 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:14,386 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:15,253 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:09:15,258 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:09:15,259 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:09:15,808 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:09:16,808 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:21,814 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:22,389 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/config.yaml 2024-04-11 12:09:26,926 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:30,254 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:09:30,258 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: stop_status 2024-04-11 12:09:30,259 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: stop_status 2024-04-11 12:09:32,310 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:34,393 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:36,289 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: pause 2024-04-11 12:09:36,290 INFO HandlerThread:472 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-11 12:09:36,290 INFO HandlerThread:472 [system_monitor.py:finish():203] Stopping system monitor 2024-04-11 12:09:36,290 INFO HandlerThread:472 [interfaces.py:finish():202] Joined cpu monitor 2024-04-11 12:09:36,291 INFO HandlerThread:472 [interfaces.py:finish():202] Joined disk monitor 2024-04-11 12:09:36,291 DEBUG SystemMonitor:472 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-11 12:09:36,291 DEBUG SystemMonitor:472 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-11 12:09:36,302 INFO HandlerThread:472 [interfaces.py:finish():202] Joined gpu monitor 2024-04-11 12:09:36,302 INFO HandlerThread:472 [interfaces.py:finish():202] Joined memory monitor 2024-04-11 12:09:36,302 INFO HandlerThread:472 [interfaces.py:finish():202] Joined network monitor 2024-04-11 12:09:36,303 DEBUG SenderThread:472 [sender.py:send():379] send: stats 2024-04-11 12:09:36,394 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:38,304 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:40,061 DEBUG SenderThread:472 [sender.py:send():379] send: exit 2024-04-11 12:09:40,061 INFO SenderThread:472 [sender.py:send_exit():586] handling exit code: 0 2024-04-11 12:09:40,061 INFO SenderThread:472 [sender.py:send_exit():588] handling runtime: 375 2024-04-11 12:09:40,061 INFO SenderThread:472 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-11 12:09:40,062 INFO SenderThread:472 [sender.py:send_exit():594] send defer 2024-04-11 12:09:40,062 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,062 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 0 2024-04-11 12:09:40,062 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,062 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 0 2024-04-11 12:09:40,062 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 1 2024-04-11 12:09:40,063 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,063 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 1 2024-04-11 12:09:40,063 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,063 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 1 2024-04-11 12:09:40,063 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 2 2024-04-11 12:09:40,063 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,063 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 2 2024-04-11 12:09:40,063 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,063 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 2 2024-04-11 12:09:40,063 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 3 2024-04-11 12:09:40,063 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,064 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 3 2024-04-11 12:09:40,064 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,064 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 3 2024-04-11 12:09:40,064 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 4 2024-04-11 12:09:40,064 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,064 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 4 2024-04-11 12:09:40,064 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,064 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 4 2024-04-11 12:09:40,064 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 5 2024-04-11 12:09:40,064 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,064 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 5 2024-04-11 12:09:40,065 DEBUG SenderThread:472 [sender.py:send():379] send: summary 2024-04-11 12:09:40,065 INFO SenderThread:472 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-11 12:09:40,065 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,065 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 5 2024-04-11 12:09:40,065 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 6 2024-04-11 12:09:40,066 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,066 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 6 2024-04-11 12:09:40,066 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,066 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 6 2024-04-11 12:09:40,066 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 7 2024-04-11 12:09:40,066 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: status_report 2024-04-11 12:09:40,066 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:40,066 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 7 2024-04-11 12:09:40,067 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:40,067 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 7 2024-04-11 12:09:40,395 INFO Thread-12 :472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-summary.json 2024-04-11 12:09:41,061 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: poll_exit 2024-04-11 12:09:41,992 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 8 2024-04-11 12:09:41,992 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: poll_exit 2024-04-11 12:09:41,992 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:41,993 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 8 2024-04-11 12:09:41,993 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:41,993 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 8 2024-04-11 12:09:41,993 INFO SenderThread:472 [job_builder.py:build():318] Attempting to build job artifact 2024-04-11 12:09:42,034 INFO SenderThread:472 [job_builder.py:_get_source_type():466] no source found 2024-04-11 12:09:42,034 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 9 2024-04-11 12:09:42,034 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,034 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 9 2024-04-11 12:09:42,035 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,035 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 9 2024-04-11 12:09:42,035 INFO SenderThread:472 [dir_watcher.py:finish():358] shutting down directory watcher 2024-04-11 12:09:42,061 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: poll_exit 2024-04-11 12:09:42,397 INFO SenderThread:472 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:42,397 INFO SenderThread:472 [dir_watcher.py:finish():388] scan: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files 2024-04-11 12:09:42,397 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-metadata.json wandb-metadata.json 2024-04-11 12:09:42,397 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/conda-environment.yaml conda-environment.yaml 2024-04-11 12:09:42,398 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log output.log 2024-04-11 12:09:42,398 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-summary.json wandb-summary.json 2024-04-11 12:09:42,402 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/config.yaml config.yaml 2024-04-11 12:09:42,406 INFO SenderThread:472 [dir_watcher.py:finish():402] scan save: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/requirements.txt requirements.txt 2024-04-11 12:09:42,411 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 10 2024-04-11 12:09:42,411 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: poll_exit 2024-04-11 12:09:42,411 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,412 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 10 2024-04-11 12:09:42,415 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,415 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 10 2024-04-11 12:09:42,415 INFO SenderThread:472 [file_pusher.py:finish():172] shutting down file pusher 2024-04-11 12:09:42,628 INFO wandb-upload_0:472 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/output.log 2024-04-11 12:09:42,629 INFO wandb-upload_2:472 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/config.yaml 2024-04-11 12:09:42,632 INFO wandb-upload_1:472 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/wandb-summary.json 2024-04-11 12:09:42,702 INFO wandb-upload_3:472 [upload_job.py:push():131] Uploaded file /kaggle/working/wandb/run-20240411_115958-sm65qpc7/files/requirements.txt 2024-04-11 12:09:42,903 INFO Thread-11 (_thread_body):472 [sender.py:transition_state():614] send defer: 11 2024-04-11 12:09:42,903 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,903 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 11 2024-04-11 12:09:42,904 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,904 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 11 2024-04-11 12:09:42,904 INFO SenderThread:472 [file_pusher.py:join():178] waiting for file pusher 2024-04-11 12:09:42,904 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 12 2024-04-11 12:09:42,904 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,904 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 12 2024-04-11 12:09:42,905 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,905 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 12 2024-04-11 12:09:42,905 INFO SenderThread:472 [file_stream.py:finish():614] file stream finish called 2024-04-11 12:09:42,989 INFO SenderThread:472 [file_stream.py:finish():618] file stream finish is done 2024-04-11 12:09:42,989 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 13 2024-04-11 12:09:42,990 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,990 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 13 2024-04-11 12:09:42,990 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,990 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 13 2024-04-11 12:09:42,990 INFO SenderThread:472 [sender.py:transition_state():614] send defer: 14 2024-04-11 12:09:42,991 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: defer 2024-04-11 12:09:42,991 INFO HandlerThread:472 [handler.py:handle_request_defer():172] handle defer: 14 2024-04-11 12:09:42,991 DEBUG SenderThread:472 [sender.py:send():379] send: final 2024-04-11 12:09:42,991 DEBUG SenderThread:472 [sender.py:send():379] send: footer 2024-04-11 12:09:42,991 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: defer 2024-04-11 12:09:42,991 INFO SenderThread:472 [sender.py:send_request_defer():610] handle sender defer: 14 2024-04-11 12:09:42,992 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: poll_exit 2024-04-11 12:09:42,992 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: poll_exit 2024-04-11 12:09:42,993 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: poll_exit 2024-04-11 12:09:42,993 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: poll_exit 2024-04-11 12:09:42,994 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: server_info 2024-04-11 12:09:42,994 DEBUG SenderThread:472 [sender.py:send_request():406] send_request: server_info 2024-04-11 12:09:42,998 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: get_summary 2024-04-11 12:09:42,999 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: sampled_history 2024-04-11 12:09:42,999 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-11 12:09:43,015 INFO MainThread:472 [wandb_run.py:_footer_history_summary_info():3920] rendering history 2024-04-11 12:09:43,015 INFO MainThread:472 [wandb_run.py:_footer_history_summary_info():3952] rendering summary 2024-04-11 12:09:43,015 INFO MainThread:472 [wandb_run.py:_footer_sync_info():3879] logging synced files 2024-04-11 12:09:43,016 DEBUG HandlerThread:472 [handler.py:handle_request():146] handle_request: shutdown 2024-04-11 12:09:43,016 INFO HandlerThread:472 [handler.py:finish():866] shutting down handler 2024-04-11 12:09:43,994 INFO WriterThread:472 [datastore.py:close():296] close: /kaggle/working/wandb/run-20240411_115958-sm65qpc7/run-sm65qpc7.wandb 2024-04-11 12:09:44,015 INFO SenderThread:472 [sender.py:finish():1546] shutting down sender 2024-04-11 12:09:44,015 INFO SenderThread:472 [file_pusher.py:finish():172] shutting down file pusher 2024-04-11 12:09:44,015 INFO SenderThread:472 [file_pusher.py:join():178] waiting for file pusher