| 2023-08-31 16:21:12,589 INFO StreamThr :3427956 [internal.py:wandb_internal():86] W&B internal server running at pid: 3427956, started at: 2023-08-31 16:21:12.587723 | |
| 2023-08-31 16:21:12,590 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status | |
| 2023-08-31 16:21:12,598 INFO WriterThread:3427956 [datastore.py:open_for_write():85] open: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb | |
| 2023-08-31 16:21:12,600 DEBUG SenderThread:3427956 [sender.py:send():369] send: header | |
| 2023-08-31 16:21:12,601 DEBUG SenderThread:3427956 [sender.py:send():369] send: run | |
| 2023-08-31 16:21:12,606 INFO SenderThread:3427956 [sender.py:_maybe_setup_resume():737] checking resume status for ckadirt/brain2music/pftwit53 | |
| 2023-08-31 16:21:13,166 INFO SenderThread:3427956 [dir_watcher.py:__init__():211] watching files in: ./wandb/run-20230831_162112-pftwit53/files | |
| 2023-08-31 16:21:13,166 INFO SenderThread:3427956 [sender.py:_start_run_threads():1100] run started: pftwit53 with start time 1693498872.59191 | |
| 2023-08-31 16:21:13,166 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: summary_record | |
| 2023-08-31 16:21:13,168 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end | |
| 2023-08-31 16:21:13,171 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-08-31 16:21:13,172 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: check_version | |
| 2023-08-31 16:21:13,241 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():31] System info init | |
| 2023-08-31 16:21:13,251 DEBUG HandlerThread:3427956 [system_info.py:__init__():46] System info init done | |
| 2023-08-31 16:21:13,251 INFO HandlerThread:3427956 [system_monitor.py:start():181] Starting system monitor | |
| 2023-08-31 16:21:13,251 INFO SystemMonitor:3427956 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-08-31 16:21:13,252 INFO HandlerThread:3427956 [system_monitor.py:probe():201] Collecting system info | |
| 2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-08-31 16:21:13,252 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started disk monitoring | |
| 2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-08-31 16:21:13,253 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started memory monitoring | |
| 2023-08-31 16:21:13,254 INFO SystemMonitor:3427956 [interfaces.py:start():190] Started network monitoring | |
| 2023-08-31 16:21:13,306 DEBUG HandlerThread:3427956 [system_info.py:probe():195] Probing system | |
| 2023-08-31 16:21:13,308 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():180] Probing git | |
| 2023-08-31 16:21:13,313 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) | |
| cmdline: git rev-parse --show-toplevel | |
| stderr: 'fatal: detected dubious ownership in repository at '/fsx' | |
| To add an exception for this directory, call: | |
| git config --global --add safe.directory /fsx' | |
| 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_probe_git():188] Probing git done | |
| 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:probe():240] Probing system done | |
| 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-1037-aws-x86_64-with-glibc2.31', 'python': '3.10.8', 'heartbeatAt': '2023-08-31T16:21:13.306106', 'startedAt': '2023-08-31T16:21:12.558810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/fsx/proj-fmri/ckadirt/b2m/src/mlpdummy.py', 'codePath': 'src/mlpdummy.py', 'git': {'remote': None, 'commit': None}, 'email': None, 'root': '/fsx/proj-fmri/ckadirt/b2m', 'host': 'ip-26-0-144-149', 'username': 'ckadirt', 'executable': '/admin/home-ckadirt/miniconda3/envs/mindeye/bin/python', 'cpu_count': 48, 'cpu_count_logical': 96, 'cpu_freq': {'current': 3002.844541666667, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2773.902, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2591.475, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 2971.069, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}, {'current': 3000.0, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 232.5829315185547, 'used': 89.25303268432617}, 'gpu': 'NVIDIA A100-SXM4-40GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}, {'name': 'NVIDIA A100-SXM4-40GB', 'memory_total': 42949672960}], 'memory': {'total': 1121.815086364746}} | |
| 2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-08-31 16:21:13,313 INFO HandlerThread:3427956 [system_monitor.py:probe():214] Publishing system info | |
| 2023-08-31 16:21:13,313 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-08-31 16:21:13,316 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment | |
| 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml | |
| 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt | |
| 2023-08-31 16:21:14,170 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json | |
| 2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_conda():86] Saving conda packages done | |
| 2023-08-31 16:21:19,578 DEBUG HandlerThread:3427956 [system_info.py:_save_code():89] Saving code | |
| 2023-08-31 16:21:19,582 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) | |
| cmdline: git rev-parse --show-toplevel | |
| stderr: 'fatal: detected dubious ownership in repository at '/fsx' | |
| To add an exception for this directory, call: | |
| git config --global --add safe.directory /fsx' | |
| 2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_code():110] Saving code done | |
| 2023-08-31 16:21:19,588 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():127] Saving git patches | |
| 2023-08-31 16:21:19,591 ERROR HandlerThread:3427956 [gitlib.py:root():89] git root error: Cmd('git') failed due to: exit code(128) | |
| cmdline: git rev-parse --show-toplevel | |
| stderr: 'fatal: detected dubious ownership in repository at '/fsx' | |
| To add an exception for this directory, call: | |
| git config --global --add safe.directory /fsx' | |
| 2023-08-31 16:21:19,600 DEBUG HandlerThread:3427956 [system_info.py:_save_patches():169] Saving git patches done | |
| 2023-08-31 16:21:19,604 INFO HandlerThread:3427956 [system_monitor.py:probe():216] Finished publishing system info | |
| 2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-08-31 16:21:19,608 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: keepalive | |
| 2023-08-31 16:21:19,608 DEBUG SenderThread:3427956 [sender.py:send():369] send: files | |
| 2023-08-31 16:21:19,608 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now | |
| 2023-08-31 16:21:19,609 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file code/src/mlpdummy.py with policy now | |
| 2023-08-31 16:21:19,616 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-08-31 16:21:19,616 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status | |
| 2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry | |
| 2023-08-31 16:21:19,994 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric | |
| 2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: telemetry | |
| 2023-08-31 16:21:19,995 DEBUG SenderThread:3427956 [sender.py:send():369] send: metric | |
| 2023-08-31 16:21:19,995 WARNING SenderThread:3427956 [sender.py:send_metric():1305] Seen metric with glob (shouldn't happen) | |
| 2023-08-31 16:21:20,181 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml | |
| 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py | |
| 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json | |
| 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code/src | |
| 2023-08-31 16:21:20,182 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_created():272] file/dir created: ./wandb/run-20230831_162112-pftwit53/files/code | |
| 2023-08-31 16:21:20,188 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/zi0muto5-wandb-metadata.json | |
| 2023-08-31 16:21:20,359 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file /tmp/tmpcnz000j2wandb/igjvcllv-code/src/mlpdummy.py | |
| 2023-08-31 16:21:22,186 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:23,961 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-08-31 16:21:26,201 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:29,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-08-31 16:21:34,329 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-08-31 16:21:34,684 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-08-31 16:21:34,684 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: stop_status | |
| 2023-08-31 16:21:36,096 DEBUG SenderThread:3427956 [sender.py:send():369] send: exit | |
| 2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():574] handling exit code: 1 | |
| 2023-08-31 16:21:36,096 INFO SenderThread:3427956 [sender.py:send_exit():576] handling runtime: 22 | |
| 2023-08-31 16:21:36,099 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end | |
| 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_exit():582] send defer | |
| 2023-08-31 16:21:36,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:36,100 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 0 | |
| 2023-08-31 16:21:36,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 0 | |
| 2023-08-31 16:21:36,100 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 1 | |
| 2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 1 | |
| 2023-08-31 16:21:36,101 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 1 | |
| 2023-08-31 16:21:36,101 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 2 | |
| 2023-08-31 16:21:36,101 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 2 | |
| 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [system_monitor.py:finish():190] Stopping system monitor | |
| 2023-08-31 16:21:36,101 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-08-31 16:21:36,101 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():159] Starting system metrics aggregation loop | |
| 2023-08-31 16:21:36,102 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():166] Finished system metrics aggregation loop | |
| 2023-08-31 16:21:36,102 DEBUG SystemMonitor:3427956 [system_monitor.py:_start():170] Publishing last batch of metrics | |
| 2023-08-31 16:21:36,277 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:36,278 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json | |
| 2023-08-31 16:21:37,227 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-08-31 16:21:37,228 INFO HandlerThread:3427956 [interfaces.py:finish():202] Joined network monitor | |
| 2023-08-31 16:21:37,228 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:37,228 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 2 | |
| 2023-08-31 16:21:37,229 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 3 | |
| 2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send():369] send: stats | |
| 2023-08-31 16:21:37,229 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:37,229 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:37,229 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 3 | |
| 2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 3 | |
| 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 4 | |
| 2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 4 | |
| 2023-08-31 16:21:37,230 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 4 | |
| 2023-08-31 16:21:37,230 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 5 | |
| 2023-08-31 16:21:37,230 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:37,230 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 5 | |
| 2023-08-31 16:21:37,231 DEBUG SenderThread:3427956 [sender.py:send():369] send: summary | |
| 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end | |
| 2023-08-31 16:21:37,233 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 5 | |
| 2023-08-31 16:21:37,233 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 6 | |
| 2023-08-31 16:21:37,233 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:37,233 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 6 | |
| 2023-08-31 16:21:37,234 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,234 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 6 | |
| 2023-08-31 16:21:37,238 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-08-31 16:21:37,283 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json | |
| 2023-08-31 16:21:37,323 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 7 | |
| 2023-08-31 16:21:37,323 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:37,323 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 7 | |
| 2023-08-31 16:21:37,324 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:37,324 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 7 | |
| 2023-08-31 16:21:38,097 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:38,288 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:38,289 INFO Thread-12 :3427956 [dir_watcher.py:_on_file_modified():289] file/dir modified: ./wandb/run-20230831_162112-pftwit53/files/config.yaml | |
| 2023-08-31 16:21:38,923 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 8 | |
| 2023-08-31 16:21:38,923 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:38,924 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:38,924 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 8 | |
| 2023-08-31 16:21:38,924 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:38,924 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 8 | |
| 2023-08-31 16:21:38,924 INFO SenderThread:3427956 [job_builder.py:build():240] Attempting to build job artifact | |
| 2023-08-31 16:21:38,926 INFO SenderThread:3427956 [job_builder.py:build():274] no source found | |
| 2023-08-31 16:21:38,926 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 9 | |
| 2023-08-31 16:21:38,927 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:38,927 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 9 | |
| 2023-08-31 16:21:38,927 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:38,927 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 9 | |
| 2023-08-31 16:21:38,927 INFO SenderThread:3427956 [dir_watcher.py:finish():359] shutting down directory watcher | |
| 2023-08-31 16:21:39,098 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:39,295 INFO SenderThread:3427956 [dir_watcher.py:finish():389] scan: ./wandb/run-20230831_162112-pftwit53/files | |
| 2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json wandb-summary.json | |
| 2023-08-31 16:21:39,296 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/requirements.txt requirements.txt | |
| 2023-08-31 16:21:39,297 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/wandb-metadata.json wandb-metadata.json | |
| 2023-08-31 16:21:39,299 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml conda-environment.yaml | |
| 2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/config.yaml config.yaml | |
| 2023-08-31 16:21:39,301 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/output.log output.log | |
| 2023-08-31 16:21:39,308 INFO SenderThread:3427956 [dir_watcher.py:finish():403] scan save: ./wandb/run-20230831_162112-pftwit53/files/code/src/mlpdummy.py code/src/mlpdummy.py | |
| 2023-08-31 16:21:39,308 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 10 | |
| 2023-08-31 16:21:39,308 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:39,309 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:39,309 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 10 | |
| 2023-08-31 16:21:39,309 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:39,309 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 10 | |
| 2023-08-31 16:21:39,309 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher | |
| 2023-08-31 16:21:39,746 INFO wandb-upload_0:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/wandb-summary.json | |
| 2023-08-31 16:21:39,814 INFO wandb-upload_1:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/requirements.txt | |
| 2023-08-31 16:21:39,848 INFO wandb-upload_2:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/conda-environment.yaml | |
| 2023-08-31 16:21:39,965 INFO wandb-upload_3:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/config.yaml | |
| 2023-08-31 16:21:40,100 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:40,100 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:40,182 INFO wandb-upload_4:3427956 [upload_job.py:push():133] Uploaded file ./wandb/run-20230831_162112-pftwit53/files/output.log | |
| 2023-08-31 16:21:40,383 INFO Thread-11 (_thread_body):3427956 [sender.py:transition_state():602] send defer: 11 | |
| 2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 11 | |
| 2023-08-31 16:21:40,384 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 11 | |
| 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher | |
| 2023-08-31 16:21:40,384 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 12 | |
| 2023-08-31 16:21:40,384 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:40,384 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 12 | |
| 2023-08-31 16:21:40,385 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:40,385 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 12 | |
| 2023-08-31 16:21:40,463 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 13 | |
| 2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:40,464 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 13 | |
| 2023-08-31 16:21:40,464 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 13 | |
| 2023-08-31 16:21:40,464 INFO SenderThread:3427956 [sender.py:transition_state():602] send defer: 14 | |
| 2023-08-31 16:21:40,464 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: defer | |
| 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: final | |
| 2023-08-31 16:21:40,465 INFO HandlerThread:3427956 [handler.py:handle_request_defer():170] handle defer: 14 | |
| 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send():369] send: footer | |
| 2023-08-31 16:21:40,465 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: defer | |
| 2023-08-31 16:21:40,465 INFO SenderThread:3427956 [sender.py:send_request_defer():598] handle sender defer: 14 | |
| 2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:40,465 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: poll_exit | |
| 2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: server_info | |
| 2023-08-31 16:21:40,466 DEBUG SenderThread:3427956 [sender.py:send_request():396] send_request: server_info | |
| 2023-08-31 16:21:40,466 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: get_summary | |
| 2023-08-31 16:21:40,468 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: sampled_history | |
| 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3464] rendering history | |
| 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_history_summary_info():3496] rendering summary | |
| 2023-08-31 16:21:40,613 INFO MainThread:3427956 [wandb_run.py:_footer_sync_info():3423] logging synced files | |
| 2023-08-31 16:21:40,614 DEBUG HandlerThread:3427956 [handler.py:handle_request():144] handle_request: shutdown | |
| 2023-08-31 16:21:40,614 INFO HandlerThread:3427956 [handler.py:finish():854] shutting down handler | |
| 2023-08-31 16:21:41,466 INFO WriterThread:3427956 [datastore.py:close():298] close: ./wandb/run-20230831_162112-pftwit53/run-pftwit53.wandb | |
| 2023-08-31 16:21:41,613 INFO SenderThread:3427956 [sender.py:finish():1526] shutting down sender | |
| 2023-08-31 16:21:41,613 INFO SenderThread:3427956 [file_pusher.py:finish():159] shutting down file pusher | |
| 2023-08-31 16:21:41,614 INFO SenderThread:3427956 [file_pusher.py:join():164] waiting for file pusher | |