| 2024-02-26 14:11:07,569 INFO StreamThr :3960 [internal.py:wandb_internal():86] W&B internal server running at pid: 3960, started at: 2024-02-26 14:11:07.564918 | |
| 2024-02-26 14:11:07,576 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-26 14:11:09,852 INFO WriterThread:3960 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/run-8c2rkhry.wandb | |
| 2024-02-26 14:11:09,855 DEBUG SenderThread:3960 [sender.py:send():382] send: header | |
| 2024-02-26 14:11:09,896 DEBUG SenderThread:3960 [sender.py:send():382] send: run | |
| 2024-02-26 14:11:10,383 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-26 14:11:10,383 INFO SenderThread:3960 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files | |
| 2024-02-26 14:11:10,384 INFO SenderThread:3960 [sender.py:_start_run_threads():1136] run started: 8c2rkhry with start time 1708956667.571442 | |
| 2024-02-26 14:11:10,384 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-26 14:11:10,733 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-26 14:11:10,760 DEBUG HandlerThread:3960 [system_info.py:__init__():27] System info init | |
| 2024-02-26 14:11:10,767 DEBUG HandlerThread:3960 [system_info.py:__init__():42] System info init done | |
| 2024-02-26 14:11:10,767 INFO HandlerThread:3960 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 14:11:10,770 INFO SystemMonitor:3960 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 14:11:10,775 INFO HandlerThread:3960 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-26 14:11:10,783 INFO SystemMonitor:3960 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 14:11:10,784 DEBUG HandlerThread:3960 [system_info.py:probe():151] Probing system | |
| 2024-02-26 14:11:10,803 INFO SystemMonitor:3960 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 14:11:10,807 INFO SystemMonitor:3960 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 14:11:10,809 DEBUG HandlerThread:3960 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-26 14:11:10,810 DEBUG HandlerThread:3960 [system_info.py:probe():199] Probing system done | |
| 2024-02-26 14:11:10,810 DEBUG HandlerThread:3960 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-26T14:11:10.784753', 'startedAt': '2024-02-26T14:11:07.493721', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Gemma-7B-Finetuning-databricks-dolly-15k-ko-Ins.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1CktAKzOMnPt5S9ITWab1_b5cPCM7ZMNM', 'host': 'e7cca09e096a', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.96010208129883}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} | |
| 2024-02-26 14:11:10,810 INFO HandlerThread:3960 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-26 14:11:10,810 INFO HandlerThread:3960 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-26 14:11:10,825 INFO SystemMonitor:3960 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 14:11:10,847 INFO SystemMonitor:3960 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 14:11:10,854 INFO HandlerThread:3960 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-26 14:11:10,871 DEBUG SenderThread:3960 [sender.py:send():382] send: files | |
| 2024-02-26 14:11:10,871 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-26 14:11:10,970 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-26 14:11:10,971 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-26 14:11:10,972 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:11:10,982 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:11:11,022 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:11:11,389 INFO Thread-12 :3960 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-metadata.json | |
| 2024-02-26 14:11:11,390 INFO Thread-12 :3960 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/requirements.txt | |
| 2024-02-26 14:11:11,483 INFO wandb-upload_0:3960 [upload_job.py:push():131] Uploaded file /tmp/tmpjuqfefllwandb/jiy701b9-wandb-metadata.json | |
| 2024-02-26 14:11:16,971 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:11:17,355 DEBUG SenderThread:3960 [sender.py:send():382] send: telemetry | |
| 2024-02-26 14:11:17,356 DEBUG SenderThread:3960 [sender.py:send():382] send: config | |
| 2024-02-26 14:11:17,357 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:11:17,355 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:17,357 DEBUG SenderThread:3960 [sender.py:send():382] send: telemetry | |
| 2024-02-26 14:11:17,358 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:11:17,358 WARNING SenderThread:3960 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 14:11:17,358 DEBUG SenderThread:3960 [sender.py:send():382] send: telemetry | |
| 2024-02-26 14:11:22,366 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:25,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:11:25,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:11:25,982 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:11:28,083 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:33,084 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:38,097 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:39,476 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/config.yaml | |
| 2024-02-26 14:11:40,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:11:40,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:11:41,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:11:44,054 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:49,055 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:54,056 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:11:55,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:11:55,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:11:56,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:12:00,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:05,030 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:10,031 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:10,848 DEBUG SystemMonitor:3960 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 14:12:10,850 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:12:10,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:12:10,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:12:11,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:12:16,011 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:21,012 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:25,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:12:25,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:12:26,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:12:26,099 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:31,100 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:36,107 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:40,851 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:12:40,966 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:12:40,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:12:41,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:12:42,093 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:47,093 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:52,094 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:12:55,967 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:12:55,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:12:56,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:12:58,035 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:03,036 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:08,037 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:10,852 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:13:10,967 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:13:10,967 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:13:11,007 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:13:13,069 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:18,070 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:20,670 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:13:20,675 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:13:20,675 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:13:20,676 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:13:20,676 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:13:20,676 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:13:20,677 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:13:20,682 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:13:20,730 INFO Thread-12 :3960 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:13:23,684 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:26,002 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:13:26,002 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:13:26,002 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:13:26,003 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:13:29,119 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:34,120 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:39,121 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:40,852 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:13:40,976 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:13:40,977 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:13:41,017 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:13:45,110 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:45,855 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/config.yaml | |
| 2024-02-26 14:13:50,227 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:55,228 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:13:55,977 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:13:55,977 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:13:56,010 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:14:01,011 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:06,012 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:10,853 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:14:10,977 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:14:10,977 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:14:11,017 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:14:11,036 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:16,037 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:21,038 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:25,977 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:14:25,978 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:14:26,018 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:14:26,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:31,060 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:36,061 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:40,854 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:14:40,977 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:14:40,978 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:14:41,018 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:14:41,103 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:46,104 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:51,105 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:55,977 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:14:55,978 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:14:56,018 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:14:57,074 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:14:57,534 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:14:57,535 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:14:57,536 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:14:57,553 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:14:58,028 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:15:02,556 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:07,557 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:10,858 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:15:10,984 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:15:10,986 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:15:11,002 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:15:13,048 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:18,049 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:23,049 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:25,984 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:15:25,985 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:15:26,025 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:15:28,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:33,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:38,066 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:40,859 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:15:40,984 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:15:40,985 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:15:41,025 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:15:44,057 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:49,058 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:54,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:15:55,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:15:55,985 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:15:56,025 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:15:59,107 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:04,108 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:09,109 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:10,860 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:16:10,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:16:10,985 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:16:11,025 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:16:15,084 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:16,124 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:16:16,126 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:16:16,128 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:16:16,134 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:16:16,511 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:16:20,138 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:25,139 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:25,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:16:25,986 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:16:26,003 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:16:31,080 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:36,081 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:40,860 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:16:40,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:16:40,985 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:16:41,027 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:16:42,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:47,059 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:52,060 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:16:55,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:16:55,986 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:16:56,026 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:16:57,093 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:02,094 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:07,094 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:10,861 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:17:10,987 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:17:10,988 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:17:11,028 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:17:12,115 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:17,117 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:22,117 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:25,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:17:25,988 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:17:26,028 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:17:27,215 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:32,216 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:37,223 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:40,862 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:17:40,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:17:40,988 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:17:41,028 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:17:43,093 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:48,094 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:53,094 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:17:55,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:17:55,990 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:17:56,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:17:58,144 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:03,145 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:08,146 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:10,863 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:18:10,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:18:10,988 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:18:11,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:18:12,970 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:18:12,971 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:18:12,973 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:18:12,979 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:18:13,779 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:18:13,983 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:18,984 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:23,985 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:25,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:18:25,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:18:26,008 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:18:29,137 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:34,138 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:39,139 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:40,864 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:18:40,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:18:40,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:18:41,030 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:18:45,058 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:50,060 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:55,061 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:18:55,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:18:55,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:18:56,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:19:01,039 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:06,040 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:10,865 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:19:10,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:19:10,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:19:11,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:19:11,043 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:16,044 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:21,044 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:25,989 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:19:25,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:19:26,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:19:26,056 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:31,057 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:36,058 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:40,866 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:19:40,988 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:19:40,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:19:41,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:19:41,062 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:46,063 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:51,064 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:19:55,989 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:19:55,989 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:19:56,029 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:19:56,100 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:01,101 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:05,876 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:20:05,879 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:20:05,879 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:20:05,885 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:20:06,026 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:20:06,888 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:10,867 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:20:11,111 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:20:11,124 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:20:11,127 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:20:11,130 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:20:12,207 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:17,209 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:22,212 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:26,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:20:26,152 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:20:26,152 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:20:26,153 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:20:27,219 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:32,219 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:37,220 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:40,868 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:20:41,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:20:41,152 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:20:41,152 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:20:41,153 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:20:42,299 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:47,299 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:52,300 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:20:56,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:20:56,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:20:56,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:20:56,154 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:20:58,277 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:03,278 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:08,279 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:10,869 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:21:11,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:21:11,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:21:11,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:21:11,154 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:21:14,269 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:19,269 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:24,270 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:26,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:21:26,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:21:26,153 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:21:26,154 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:21:28,588 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:21:28,591 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:21:28,592 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:21:28,598 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:21:29,292 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:21:29,599 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:34,600 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:34,743 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 14:21:34,745 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:21:34,746 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:21:34,747 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:21:34,747 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:21:34,748 DEBUG SenderThread:3960 [sender.py:send():382] send: metric | |
| 2024-02-26 14:21:34,748 DEBUG SenderThread:3960 [sender.py:send():382] send: history | |
| 2024-02-26 14:21:34,755 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 14:21:34,762 INFO SenderThread:3960 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 14:21:35,308 INFO Thread-12 :3960 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko-Ins/wandb/run-20240226_141107-8c2rkhry/files/wandb-summary.json | |
| 2024-02-26 14:21:39,765 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:40,870 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:21:41,112 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 14:21:41,113 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 14:21:41,121 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 14:21:41,122 DEBUG SenderThread:3960 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 14:21:44,025 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 14:21:44,026 INFO HandlerThread:3960 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 14:21:44,026 INFO HandlerThread:3960 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 14:21:44,026 DEBUG SystemMonitor:3960 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 14:21:44,028 DEBUG SystemMonitor:3960 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 14:21:44,029 INFO HandlerThread:3960 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 14:21:44,033 INFO HandlerThread:3960 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 14:21:44,043 INFO HandlerThread:3960 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 14:21:44,044 INFO HandlerThread:3960 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 14:21:44,044 INFO HandlerThread:3960 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 14:21:44,045 DEBUG SenderThread:3960 [sender.py:send():382] send: stats | |
| 2024-02-26 14:21:45,046 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:50,047 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 14:21:50,237 DEBUG HandlerThread:3960 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 14:21:50,238 INFO HandlerThread:3960 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 14:21:50,238 INFO HandlerThread:3960 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 14:21:50,239 INFO SystemMonitor:3960 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 14:21:50,242 INFO SystemMonitor:3960 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 14:21:50,244 INFO SystemMonitor:3960 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 14:21:50,245 INFO SystemMonitor:3960 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 14:21:50,261 INFO SystemMonitor:3960 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 14:21:50,262 INFO SystemMonitor:3960 [interfaces.py:start():190] Started network monitoring | |