2024-04-09 22:14:59,283 INFO StreamThr :619 [internal.py:wandb_internal():86] W&B internal server running at pid: 619, started at: 2024-04-09 22:14:59.282742 2024-04-09 22:14:59,285 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status 2024-04-09 22:14:59,558 INFO WriterThread:619 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/run-bis0oi0s.wandb 2024-04-09 22:14:59,559 DEBUG SenderThread:619 [sender.py:send():379] send: header 2024-04-09 22:14:59,561 DEBUG SenderThread:619 [sender.py:send():379] send: run 2024-04-09 22:14:59,778 INFO SenderThread:619 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files 2024-04-09 22:14:59,778 INFO SenderThread:619 [sender.py:_start_run_threads():1124] run started: bis0oi0s with start time 1712700899.282634 2024-04-09 22:14:59,786 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: check_version 2024-04-09 22:14:59,787 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: check_version 2024-04-09 22:14:59,885 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: run_start 2024-04-09 22:14:59,895 DEBUG HandlerThread:619 [system_info.py:__init__():26] System info init 2024-04-09 22:14:59,895 DEBUG HandlerThread:619 [system_info.py:__init__():41] System info init done 2024-04-09 22:14:59,896 INFO HandlerThread:619 [system_monitor.py:start():194] Starting system monitor 2024-04-09 22:14:59,896 INFO SystemMonitor:619 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-09 22:14:59,896 INFO HandlerThread:619 [system_monitor.py:probe():214] Collecting system info 2024-04-09 22:14:59,896 INFO SystemMonitor:619 [interfaces.py:start():190] Started cpu monitoring 2024-04-09 22:14:59,897 INFO SystemMonitor:619 [interfaces.py:start():190] Started disk monitoring 2024-04-09 22:14:59,899 INFO SystemMonitor:619 [interfaces.py:start():190] Started gpu monitoring 2024-04-09 22:14:59,900 INFO SystemMonitor:619 [interfaces.py:start():190] Started memory monitoring 2024-04-09 22:14:59,901 INFO SystemMonitor:619 [interfaces.py:start():190] Started network monitoring 2024-04-09 22:14:59,915 DEBUG HandlerThread:619 [system_info.py:probe():150] Probing system 2024-04-09 22:14:59,917 DEBUG HandlerThread:619 [gitlib.py:_init_repo():56] git repository is invalid 2024-04-09 22:14:59,917 DEBUG HandlerThread:619 [system_info.py:probe():198] Probing system done 2024-04-09 22:14:59,917 DEBUG HandlerThread:619 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-09T22:14:59.915306', 'startedAt': '2024-04-09T22:14:59.276229', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '6e44b39f6877', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.152, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}, {'current': 2000.152, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5569.839126586914}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} 2024-04-09 22:14:59,917 INFO HandlerThread:619 [system_monitor.py:probe():224] Finished collecting system info 2024-04-09 22:14:59,917 INFO HandlerThread:619 [system_monitor.py:probe():227] Publishing system info 2024-04-09 22:14:59,917 DEBUG HandlerThread:619 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-04-09 22:15:00,780 INFO Thread-12 :619 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/conda-environment.yaml 2024-04-09 22:15:14,931 ERROR HandlerThread:619 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-04-09 22:15:14,932 DEBUG HandlerThread:619 [system_info.py:_save_conda():222] Saving conda packages done 2024-04-09 22:15:14,932 INFO HandlerThread:619 [system_monitor.py:probe():229] Finished publishing system info 2024-04-09 22:15:14,937 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:14,938 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: keepalive 2024-04-09 22:15:14,938 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:14,938 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: keepalive 2024-04-09 22:15:14,938 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:14,938 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: keepalive 2024-04-09 22:15:14,938 DEBUG SenderThread:619 [sender.py:send():379] send: files 2024-04-09 22:15:14,939 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now 2024-04-09 22:15:15,258 INFO wandb-upload_0:619 [upload_job.py:push():131] Uploaded file /tmp/tmpkq40x4b7wandb/n24vez5w-wandb-metadata.json 2024-04-09 22:15:15,783 INFO Thread-12 :619 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/wandb-metadata.json 2024-04-09 22:15:15,887 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: python_packages 2024-04-09 22:15:15,888 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: python_packages 2024-04-09 22:15:15,892 DEBUG SenderThread:619 [sender.py:send():379] send: telemetry 2024-04-09 22:15:15,902 DEBUG SenderThread:619 [sender.py:send():379] send: config 2024-04-09 22:15:15,904 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:15:15,904 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:15:15,905 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:15:15,906 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:15:16,034 DEBUG SenderThread:619 [sender.py:send():379] send: telemetry 2024-04-09 22:15:16,034 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:15:16,034 WARNING SenderThread:619 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-04-09 22:15:16,035 DEBUG SenderThread:619 [sender.py:send():379] send: telemetry 2024-04-09 22:15:16,784 INFO Thread-12 :619 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/output.log 2024-04-09 22:15:16,784 INFO Thread-12 :619 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/requirements.txt 2024-04-09 22:15:18,785 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/output.log 2024-04-09 22:15:20,073 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:25,074 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:30,080 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:30,789 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/config.yaml 2024-04-09 22:15:30,890 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:15:30,891 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:15:30,891 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:15:36,043 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:41,044 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:45,889 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:15:45,889 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:15:45,931 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:15:46,052 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:51,053 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:56,054 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:15:59,901 DEBUG SystemMonitor:619 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-09 22:15:59,903 DEBUG SenderThread:619 [sender.py:send():379] send: stats 2024-04-09 22:16:00,889 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:16:00,890 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:16:00,930 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:16:02,049 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:07,050 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:12,051 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:14,798 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: partial_history 2024-04-09 22:16:14,800 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:16:14,801 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:16:14,801 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:16:14,801 DEBUG SenderThread:619 [sender.py:send():379] send: metric 2024-04-09 22:16:14,801 DEBUG SenderThread:619 [sender.py:send():379] send: history 2024-04-09 22:16:14,801 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:16:14,803 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:16:14,806 INFO Thread-12 :619 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/wandb-summary.json 2024-04-09 22:16:15,892 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:16:15,892 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:16:15,893 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:16:16,807 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/output.log 2024-04-09 22:16:17,991 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:22,991 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:27,992 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:29,904 DEBUG SenderThread:619 [sender.py:send():379] send: stats 2024-04-09 22:16:30,889 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:16:30,890 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:16:30,930 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:16:33,014 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:33,814 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/config.yaml 2024-04-09 22:16:38,152 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:43,153 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:45,889 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:16:45,890 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:16:45,930 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:16:49,004 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:54,005 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:59,006 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:16:59,905 DEBUG SenderThread:619 [sender.py:send():379] send: stats 2024-04-09 22:17:00,890 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: stop_status 2024-04-09 22:17:00,890 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: stop_status 2024-04-09 22:17:00,931 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-09 22:17:04,047 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report 2024-04-09 22:17:08,479 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: partial_history 2024-04-09 22:17:08,480 DEBUG SenderThread:619 [sender.py:send():379] send: history 2024-04-09 22:17:08,480 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,481 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,828 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/wandb-summary.json 2024-04-09 22:17:08,906 DEBUG SenderThread:619 [sender.py:send():379] send: telemetry 2024-04-09 22:17:08,907 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,909 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: partial_history 2024-04-09 22:17:08,910 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,911 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,911 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,911 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,912 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,912 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,912 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,913 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,913 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:08,913 DEBUG SenderThread:619 [sender.py:send():379] send: history 2024-04-09 22:17:08,914 DEBUG SenderThread:619 [sender.py:send_request():406] send_request: summary_record 2024-04-09 22:17:08,914 INFO SenderThread:619 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-09 22:17:09,828 INFO Thread-12 :619 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240409_221459-bis0oi0s/files/wandb-summary.json 2024-04-09 22:17:09,914 DEBUG HandlerThread:619 [handler.py:handle_request():146] handle_request: status_report