2024-02-23 03:24:22,985 INFO StreamThr :116329 [internal.py:wandb_internal():86] W&B internal server running at pid: 116329, started at: 2024-02-23 03:24:22.985254 2024-02-23 03:24:22,987 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status 2024-02-23 03:24:22,988 INFO WriterThread:116329 [datastore.py:open_for_write():87] open: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/run-b657btrg.wandb 2024-02-23 03:24:22,988 DEBUG SenderThread:116329 [sender.py:send():382] send: header 2024-02-23 03:24:22,988 DEBUG SenderThread:116329 [sender.py:send():382] send: run 2024-02-23 03:24:23,687 INFO SenderThread:116329 [dir_watcher.py:__init__():211] watching files in: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files 2024-02-23 03:24:23,687 INFO SenderThread:116329 [sender.py:_start_run_threads():1136] run started: b657btrg with start time 1708638862.984978 2024-02-23 03:24:23,690 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: check_version 2024-02-23 03:24:23,691 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: check_version 2024-02-23 03:24:23,801 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: run_start 2024-02-23 03:24:23,806 DEBUG HandlerThread:116329 [system_info.py:__init__():27] System info init 2024-02-23 03:24:23,806 DEBUG HandlerThread:116329 [system_info.py:__init__():42] System info init done 2024-02-23 03:24:23,806 INFO HandlerThread:116329 [system_monitor.py:start():194] Starting system monitor 2024-02-23 03:24:23,806 INFO SystemMonitor:116329 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-23 03:24:23,807 INFO HandlerThread:116329 [system_monitor.py:probe():214] Collecting system info 2024-02-23 03:24:23,807 INFO SystemMonitor:116329 [interfaces.py:start():190] Started cpu monitoring 2024-02-23 03:24:23,808 INFO SystemMonitor:116329 [interfaces.py:start():190] Started disk monitoring 2024-02-23 03:24:23,808 INFO SystemMonitor:116329 [interfaces.py:start():190] Started gpu monitoring 2024-02-23 03:24:23,809 INFO SystemMonitor:116329 [interfaces.py:start():190] Started memory monitoring 2024-02-23 03:24:23,809 INFO SystemMonitor:116329 [interfaces.py:start():190] Started network monitoring 2024-02-23 03:24:23,837 DEBUG HandlerThread:116329 [system_info.py:probe():151] Probing system 2024-02-23 03:24:23,840 DEBUG HandlerThread:116329 [gitlib.py:_init_repo():56] git repository is invalid 2024-02-23 03:24:23,840 DEBUG HandlerThread:116329 [system_info.py:probe():199] Probing system done 2024-02-23 03:24:23,840 DEBUG HandlerThread:116329 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-172-generic-x86_64-with-glibc2.31', 'python': '3.9.18', 'heartbeatAt': '2024-02-22T21:54:23.837753', 'startedAt': '2024-02-22T21:54:22.976651', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '/home/rnd/Documents/Ameer/gemma/new.py', 'codePathLocal': 'new.py', 'codePath': 'new.py', 'host': 'rnd-System-Product-Name', 'username': 'rnd', 'executable': '/home/rnd/miniconda3/envs/gemma/bin/python', 'cpu_count': 24, 'cpu_count_logical': 32, 'cpu_freq': {'current': 4938.592250000001, 'min': 800.0, 'max': 5700.0}, 'cpu_freq_per_core': [{'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5504.691, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5526.894, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5516.923, 'min': 800.0, 'max': 7000.0}, {'current': 5800.0, 'min': 800.0, 'max': 7400.0}, {'current': 5819.727, 'min': 800.0, 'max': 7400.0}, {'current': 5684.191, 'min': 800.0, 'max': 7400.0}, {'current': 5815.223, 'min': 800.0, 'max': 7400.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5500.0, 'min': 800.0, 'max': 7000.0}, {'current': 5536.423, 'min': 800.0, 'max': 7000.0}, {'current': 4296.422, 'min': 800.0, 'max': 4300.0}, {'current': 4302.26, 'min': 800.0, 'max': 4300.0}, {'current': 4292.475, 'min': 800.0, 'max': 4300.0}, {'current': 4279.054, 'min': 800.0, 'max': 4300.0}, {'current': 4283.433, 'min': 800.0, 'max': 4300.0}, {'current': 4300.044, 'min': 800.0, 'max': 4300.0}, {'current': 4368.421, 'min': 800.0, 'max': 4300.0}, {'current': 4280.523, 'min': 800.0, 'max': 4300.0}, {'current': 4291.353, 'min': 800.0, 'max': 4300.0}, {'current': 4314.482, 'min': 800.0, 'max': 4300.0}, {'current': 4299.578, 'min': 800.0, 'max': 4300.0}, {'current': 4300.072, 'min': 800.0, 'max': 4300.0}, {'current': 4301.981, 'min': 800.0, 'max': 4300.0}, {'current': 4311.285, 'min': 800.0, 'max': 4300.0}, {'current': 4302.597, 'min': 800.0, 'max': 4300.0}, {'current': 4306.9, 'min': 800.0, 'max': 4300.0}], 'disk': {'/': {'total': 1832.2072448730469, 'used': 1698.8227272033691}}, 'gpu': 'NVIDIA GeForce RTX 3090 Ti', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 3090 Ti', 'memory_total': 25757220864}], 'memory': {'total': 62.508731842041016}} 2024-02-23 03:24:23,840 INFO HandlerThread:116329 [system_monitor.py:probe():224] Finished collecting system info 2024-02-23 03:24:23,840 INFO HandlerThread:116329 [system_monitor.py:probe():227] Publishing system info 2024-02-23 03:24:23,840 DEBUG HandlerThread:116329 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment 2024-02-23 03:24:24,688 INFO Thread-12 :116329 [dir_watcher.py:_on_file_created():271] file/dir created: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/conda-environment.yaml 2024-02-23 03:24:24,865 DEBUG HandlerThread:116329 [system_info.py:_save_conda():220] Saving conda packages done 2024-02-23 03:24:24,866 INFO HandlerThread:116329 [system_monitor.py:probe():229] Finished publishing system info 2024-02-23 03:24:24,867 DEBUG SenderThread:116329 [sender.py:send():382] send: files 2024-02-23 03:24:24,868 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-02-23 03:24:24,870 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: python_packages 2024-02-23 03:24:24,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:24:24,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: python_packages 2024-02-23 03:24:24,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:24:24,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: telemetry 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: config 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: telemetry 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:25,230 WARNING SenderThread:116329 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-02-23 03:24:25,230 DEBUG SenderThread:116329 [sender.py:send():382] send: telemetry 2024-02-23 03:24:25,687 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/conda-environment.yaml 2024-02-23 03:24:25,687 INFO Thread-12 :116329 [dir_watcher.py:_on_file_created():271] file/dir created: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:25,688 INFO Thread-12 :116329 [dir_watcher.py:_on_file_created():271] file/dir created: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-metadata.json 2024-02-23 03:24:25,688 INFO Thread-12 :116329 [dir_watcher.py:_on_file_created():271] file/dir created: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/requirements.txt 2024-02-23 03:24:26,812 INFO wandb-upload_0:116329 [upload_job.py:push():131] Uploaded file /tmp/tmpdnshe0bdwandb/7ag27ml8-wandb-metadata.json 2024-02-23 03:24:27,688 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:28,103 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:28,702 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:24:28,703 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:24:28,704 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:24:29,688 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:29,689 INFO Thread-12 :116329 [dir_watcher.py:_on_file_created():271] file/dir created: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:24:31,689 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:33,147 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:33,689 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:35,690 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:37,690 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:38,209 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:39,691 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:39,870 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:24:39,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:24:39,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:24:41,691 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:43,267 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:43,692 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:45,692 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:47,693 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:48,342 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:49,693 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:51,694 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:53,418 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:53,694 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:54,695 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/config.yaml 2024-02-23 03:24:54,870 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:24:54,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:24:54,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:24:55,695 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:57,695 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:24:58,948 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:24:59,696 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:01,696 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:03,697 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:04,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:05,697 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:07,698 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:09,097 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:09,698 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:09,870 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:25:09,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:25:09,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:25:11,699 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:13,699 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:14,170 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:15,700 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:17,700 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:19,249 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:19,701 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:21,701 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:23,702 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:23,810 DEBUG SystemMonitor:116329 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-23 03:25:23,811 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:25:24,331 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:24,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:25:24,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:25:24,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:25:25,702 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:27,703 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:29,411 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:29,703 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:31,704 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:33,704 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:34,489 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:35,705 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:37,705 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:39,567 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:39,706 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:39,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:25:39,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:25:39,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:25:41,706 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:43,707 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:44,650 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:45,707 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:47,708 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:49,708 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:49,732 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:51,709 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:53,709 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:53,811 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:25:54,815 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:25:54,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:25:54,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:25:54,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:25:55,710 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:57,710 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:59,711 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:25:59,922 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:01,711 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:03,712 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:05,028 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:05,712 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:07,713 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:09,713 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:09,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:26:09,871 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:26:09,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:26:10,191 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:11,714 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:13,714 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:15,225 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:15,715 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:17,715 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:19,716 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:20,315 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:21,716 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:23,717 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:23,812 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:26:24,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:26:24,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:26:24,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:26:25,410 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:25,717 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:27,718 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:29,718 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:30,501 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:31,719 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:33,719 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:35,596 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:35,720 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:37,720 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:39,721 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:39,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:26:39,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:26:39,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:26:40,690 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:41,721 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:43,722 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:45,722 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:45,784 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:47,723 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:49,723 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:50,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:51,724 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:53,724 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:53,812 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:26:54,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:26:54,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:26:54,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:26:55,725 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:55,970 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:26:57,725 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:26:59,726 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:01,065 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:01,726 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:03,727 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:05,727 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:06,160 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:07,728 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:09,728 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:09,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:27:09,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:27:09,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:27:11,259 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:11,729 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:13,729 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:15,729 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:16,352 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:17,730 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:19,730 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:21,447 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:21,731 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:23,731 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:23,813 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:27:24,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:27:24,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:27:24,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:27:25,732 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:26,539 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:27,733 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:29,734 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:31,631 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:31,734 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:33,735 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:35,735 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:36,725 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:37,736 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:39,736 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:39,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:27:39,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:27:39,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:27:41,737 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:41,821 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:43,737 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:45,738 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:46,915 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:47,738 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:49,739 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:51,739 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:52,011 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:53,740 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:53,814 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:27:54,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:27:54,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:27:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:27:55,740 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:57,104 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:27:57,741 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:27:59,741 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:01,742 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:02,205 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:03,742 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:05,743 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:07,315 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:07,743 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:09,744 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:09,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:28:09,872 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:28:09,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:28:11,744 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:12,427 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:13,745 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:15,745 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:17,538 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:17,746 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:19,746 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:21,746 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:22,654 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:23,747 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:23,814 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:28:24,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:28:24,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:28:24,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:28:25,747 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:27,748 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:27,765 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:29,748 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:31,749 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:32,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:33,749 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:35,750 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:37,750 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:37,992 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:39,751 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:39,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:28:39,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:28:39,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:28:41,751 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:43,103 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:43,752 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:45,752 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:47,753 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:48,218 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:49,753 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:51,754 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:53,327 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:53,754 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:53,815 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:28:54,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:28:54,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:28:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:28:55,755 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:57,755 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:28:58,443 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:28:59,756 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:01,756 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:03,552 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:03,757 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:05,757 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:07,758 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:08,664 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:09,758 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:09,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:29:09,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:29:09,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:29:11,759 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:13,759 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:13,771 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:15,760 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:17,760 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:18,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:19,761 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:21,761 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:23,762 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:23,815 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:29:23,989 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:24,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:29:24,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:29:24,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:29:25,762 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:27,763 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:29,101 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:29,763 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:31,764 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:33,764 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:34,211 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:35,765 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:37,765 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:39,323 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:39,766 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:39,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:29:39,873 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:29:39,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:29:41,766 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:43,767 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:44,430 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:45,767 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:47,768 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:49,540 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:49,769 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:51,769 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:53,770 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:53,816 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:29:54,646 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:29:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:29:54,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:29:55,770 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:57,771 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:29:59,756 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:29:59,771 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:01,771 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:03,772 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:04,067 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send():382] send: metric 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:30:04,068 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:30:04,069 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:30:04,772 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:30:05,069 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:05,772 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:07,773 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:09,663 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:30:09,664 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:30:09,664 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:30:09,664 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:30:09,773 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:09,774 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:30:09,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:30:09,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:30:09,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:30:10,239 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:11,774 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:13,774 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:15,320 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:15,775 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:17,776 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:19,776 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:20,428 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:21,777 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:23,777 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:23,817 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:30:24,873 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:30:24,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:30:24,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:30:25,537 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:25,778 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:27,778 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:29,779 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:30,649 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:31,779 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:32,780 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/config.yaml 2024-02-23 03:30:33,780 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:35,780 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:37,733 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:37,781 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:39,781 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:39,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:30:39,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:30:39,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:30:41,782 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:42,842 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:43,782 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:45,783 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:47,783 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:47,955 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:49,784 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:51,784 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:53,069 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:53,785 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:53,817 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:30:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:30:54,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:30:54,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:30:55,785 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:57,786 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:30:58,184 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:30:59,786 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:01,787 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:03,296 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:03,787 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:05,788 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:07,789 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:08,306 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:09,789 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:09,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:31:09,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:31:09,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:31:11,790 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:13,310 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:13,790 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:15,791 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:17,791 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:18,314 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:19,792 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:21,792 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:23,326 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:23,793 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:23,818 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:31:24,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:31:24,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:31:24,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:31:25,793 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:27,794 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:28,436 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:29,794 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:31,795 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:33,544 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:33,795 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:35,796 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:37,796 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:38,655 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:39,797 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:39,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:31:39,874 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:31:39,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:31:41,797 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:43,767 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:43,798 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:45,798 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:47,799 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:48,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:49,799 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:51,800 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:53,800 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:53,818 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:31:53,987 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:54,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:31:54,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:31:54,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:31:55,801 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:57,801 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:31:58,989 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:31:59,802 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:01,802 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:03,803 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:04,099 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:05,804 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:08,805 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:09,210 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:09,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:32:09,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:32:09,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:32:10,805 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:12,806 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:14,318 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:14,806 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:16,807 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:18,807 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:19,430 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:20,807 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:22,808 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:23,819 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:32:24,538 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:24,808 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:24,874 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:32:24,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:32:24,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:32:26,809 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:28,809 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:29,649 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:30,810 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:32,810 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:34,758 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:34,811 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:36,811 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:38,812 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:39,868 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:39,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:32:39,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:32:39,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:32:40,812 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:42,813 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:44,813 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:44,976 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:46,814 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:48,814 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:50,088 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:50,815 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:52,815 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:53,820 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:32:54,816 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:54,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:32:54,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:32:54,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:32:55,192 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:32:56,816 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:32:58,817 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:00,198 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:00,817 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:02,818 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:04,818 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:05,308 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:06,819 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:08,819 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:09,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:33:09,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:33:09,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:33:10,421 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:10,820 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:12,820 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:14,821 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:15,528 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:16,821 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:18,822 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:20,639 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:20,822 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:22,823 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:23,820 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:33:24,823 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:24,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:33:24,875 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:33:24,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:33:25,749 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:26,824 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:28,824 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:30,825 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:30,858 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:32,825 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:34,826 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:35,973 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:36,826 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:38,827 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:39,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:33:39,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:33:39,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:33:40,827 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:41,082 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:42,828 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:44,829 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:46,190 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:46,829 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:48,830 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:50,830 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:51,297 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:52,831 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:53,821 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:33:54,831 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:54,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:33:54,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:33:54,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:33:56,403 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:33:56,832 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:33:58,832 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:00,833 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:01,510 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:02,833 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:04,834 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:06,616 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:06,835 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:08,835 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:09,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:34:09,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:34:09,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:34:10,836 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:11,727 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:12,836 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:14,837 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:16,837 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:16,837 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:18,838 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:20,838 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:21,944 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:22,839 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:23,821 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:34:24,839 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:24,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:34:24,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:34:24,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:34:26,840 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:27,052 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:28,840 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:30,841 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:32,161 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:32,841 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:34,842 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:36,843 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:37,266 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:38,843 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:39,875 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:34:39,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:34:39,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:34:40,844 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:42,371 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:42,844 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:44,845 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:46,845 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:47,477 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:48,846 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:50,846 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:52,583 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:52,847 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:53,822 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:34:54,847 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:54,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:34:54,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:34:54,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:34:56,848 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:34:57,689 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:34:58,848 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:00,849 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:02,794 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:02,849 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:04,850 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:06,850 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:07,894 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:08,851 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:09,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:35:09,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:35:09,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:35:10,851 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:12,852 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:13,000 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:14,852 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:16,853 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:18,104 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:18,853 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:20,854 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:22,854 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:23,216 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:23,823 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:35:24,855 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:24,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:35:24,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:35:24,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:35:26,856 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:28,318 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:28,856 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:30,857 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:32,857 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:33,422 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:34,858 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:36,858 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:38,525 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:38,859 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:39,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:35:39,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:35:39,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:35:40,859 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:42,860 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:43,629 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:44,860 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:45,966 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:35:45,967 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:35:45,967 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:35:45,967 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:35:46,861 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:46,861 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:35:47,861 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:48,720 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:48,861 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:50,862 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:51,586 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:35:51,587 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:35:51,587 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:35:51,587 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:35:51,862 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:35:52,862 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:53,754 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:53,823 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:35:53,863 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:54,863 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:54,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:35:54,876 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:35:54,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:35:56,864 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:35:58,855 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:35:58,864 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:00,865 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:02,865 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:03,960 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:04,866 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:06,866 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:08,867 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:09,061 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:09,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:36:09,880 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:36:09,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:36:10,867 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:12,868 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:14,165 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:14,868 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:16,869 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:18,869 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:19,269 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:20,870 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:22,870 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:23,824 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:36:24,372 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:24,871 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:24,876 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:36:24,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:36:24,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:36:26,871 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:28,872 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:29,473 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:30,872 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:32,873 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:34,575 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:34,873 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:36,874 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:38,874 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:39,678 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:39,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:36:39,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:36:39,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:36:40,875 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:42,875 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:44,778 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:44,876 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:46,876 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:48,877 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:49,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:50,878 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:52,878 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:53,824 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:36:54,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:36:54,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:36:54,879 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:54,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:36:55,179 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:36:56,879 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:36:58,880 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:00,198 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:00,880 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:02,881 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:04,881 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:05,297 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:06,882 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:08,882 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:09,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:37:09,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:37:09,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:37:10,398 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:10,883 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:12,883 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:14,884 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:15,496 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:16,884 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:18,885 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:20,598 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:20,885 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:22,886 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:23,825 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:37:24,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:37:24,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:37:24,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:37:24,886 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:25,642 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:26,887 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:28,887 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:30,745 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:30,888 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:32,888 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:34,889 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:35,816 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:36,890 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:38,890 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:39,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:37:39,877 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:37:39,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:37:40,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:40,891 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:42,891 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:44,892 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:37:45,992 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:50,993 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:37:53,826 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:37:54,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:37:54,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:37:54,880 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:37:56,242 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:01,244 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:06,244 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:09,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:38:09,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:38:09,919 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:38:11,351 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:12,903 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:14,904 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:16,422 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:16,905 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:18,905 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:20,906 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:21,487 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:22,906 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:23,826 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:38:24,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:38:24,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:38:24,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:38:24,907 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:26,539 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:26,907 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:28,908 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:30,908 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:31,591 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:32,909 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:34,909 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:36,649 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:36,910 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:38,910 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:39,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:38:39,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:38:39,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:38:40,911 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:41,702 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:42,911 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:44,912 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:46,766 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:46,912 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:48,913 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:50,913 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:51,834 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:52,914 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:53,827 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:38:54,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:38:54,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:38:54,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:38:54,914 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:56,900 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:38:56,915 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:38:58,915 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:00,916 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:01,968 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:02,916 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:04,917 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:06,917 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:07,035 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:08,918 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:09,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:39:09,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:39:09,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:39:10,918 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:12,103 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:12,919 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:14,919 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:16,920 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:17,170 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:18,920 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:20,921 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:22,237 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:22,921 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:23,827 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:39:24,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:39:24,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:39:24,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:39:24,922 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:26,922 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:27,305 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:28,923 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:30,923 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:32,373 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:32,924 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:34,924 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:36,925 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:37,440 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:38,925 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:39,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:39:39,878 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:39:39,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:39:40,925 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:42,507 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:42,926 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:44,926 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:46,927 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:47,574 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:48,927 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:50,927 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:52,644 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:52,928 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:53,828 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:39:54,878 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:39:54,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:39:54,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:39:54,928 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:56,928 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:39:57,713 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:39:58,929 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:00,929 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:02,784 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:02,930 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:04,930 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:06,931 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:07,852 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:08,931 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:09,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:40:09,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:40:09,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:40:10,931 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:12,923 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:12,932 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:14,932 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:16,932 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:17,990 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:18,933 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:20,933 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:22,934 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:23,057 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:23,828 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:40:24,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:40:24,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:40:24,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:40:24,934 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:26,934 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:28,171 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:28,935 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:30,935 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:32,935 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:33,288 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:34,936 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:36,937 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:38,376 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:38,937 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:39,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:40:39,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:40:39,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:40:40,938 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:42,938 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:43,443 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:44,938 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:46,939 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:48,510 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:48,939 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:50,940 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:52,940 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:53,578 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:53,829 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:40:54,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:40:54,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:40:54,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:40:54,940 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:56,941 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:40:58,615 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:40:58,941 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:00,942 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:02,942 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:03,683 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:04,942 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:06,943 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:08,707 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:08,943 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:09,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:41:09,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:41:09,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:41:10,944 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:12,944 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:13,772 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:14,944 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:16,945 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:18,843 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:18,945 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:20,946 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:22,946 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:23,830 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:41:23,916 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:24,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:41:24,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:41:24,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:41:24,946 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:26,947 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:28,947 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:28,984 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:30,948 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:32,948 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:34,053 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:34,948 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:36,949 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:38,949 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:39,065 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:39,879 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:41:39,879 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:41:39,882 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:41:40,950 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:42,950 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:44,150 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:44,950 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:46,951 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:48,951 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:49,217 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:50,952 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:52,952 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:53,830 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:41:53,952 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:41:53,953 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:41:53,953 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:41:53,953 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:41:54,953 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:54,953 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:41:54,954 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:41:55,015 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:41:55,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:41:55,016 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:41:55,953 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:56,953 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:58,954 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:41:59,550 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:41:59,551 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:41:59,551 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:41:59,552 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:41:59,954 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:41:59,998 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:00,954 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:01,954 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:02,955 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:04,955 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:05,004 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:06,956 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:08,956 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:10,015 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:42:10,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:10,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:42:10,016 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:42:10,957 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:12,957 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:14,958 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:15,126 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:16,958 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:18,959 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:20,195 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:20,959 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:22,960 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:23,831 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:42:24,960 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:25,015 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:42:25,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:42:25,016 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:42:25,326 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:26,961 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:28,961 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:30,331 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:30,962 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:32,962 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:34,963 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:35,399 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:36,963 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:38,964 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:40,015 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:42:40,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:42:40,016 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:42:40,467 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:40,964 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:42,965 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:44,965 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:45,529 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:46,966 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:48,966 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:50,612 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:50,967 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:52,967 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:53,831 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:42:54,967 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:55,015 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:42:55,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:42:55,016 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:42:55,716 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:42:56,968 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:42:58,969 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:00,788 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:00,969 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:02,970 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:04,970 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:05,804 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:06,970 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:08,971 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:10,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:43:10,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:43:10,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:43:10,838 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:10,971 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:12,972 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:14,972 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:15,905 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:16,972 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:18,973 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:20,973 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:20,974 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:22,974 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:23,832 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:43:24,975 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:25,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:43:25,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:43:25,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:43:26,040 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:26,975 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:28,975 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:30,976 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:31,109 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:32,976 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:34,977 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:36,183 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:36,977 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:38,977 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:40,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:43:40,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:43:40,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:43:40,978 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:41,251 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:42,978 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:44,978 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:46,266 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:46,979 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:48,979 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:50,980 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:51,349 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:52,980 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:53,833 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:43:54,981 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:55,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:43:55,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:43:55,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:43:56,376 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:43:56,981 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:43:58,982 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:00,982 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:01,450 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:02,983 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:04,983 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:06,519 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:06,984 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:08,985 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:10,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:44:10,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:44:10,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:44:10,985 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:11,528 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:12,987 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:14,987 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:16,586 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:16,988 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:18,988 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:20,989 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:21,665 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:22,989 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:23,834 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:44:24,990 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:25,016 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:44:25,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:44:25,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:44:26,735 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:26,990 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:28,991 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:30,991 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:31,802 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:32,992 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:34,992 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:36,869 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:36,993 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:38,993 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:40,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:44:40,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:44:40,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:44:40,994 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:41,937 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:42,994 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:44,995 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:46,995 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:47,005 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:48,995 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:50,996 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:52,074 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:52,996 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:53,834 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:44:54,997 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:55,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:44:55,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:44:55,017 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:44:56,998 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:44:57,131 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:44:58,998 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:00,999 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:02,147 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:02,999 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:05,000 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:07,000 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:07,209 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:09,001 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:10,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:45:10,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:45:10,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:45:11,001 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:12,277 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:13,001 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:15,002 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:17,003 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:17,343 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:19,003 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:21,004 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:22,411 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:23,004 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:23,834 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:45:25,005 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:25,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:45:25,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:45:25,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:45:27,005 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:27,478 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:29,006 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:31,006 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:32,543 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:33,007 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:35,007 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:37,008 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:37,608 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:39,008 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:40,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:45:40,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:45:40,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:45:41,009 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:42,674 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:43,009 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:45,010 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:47,010 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:47,740 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:49,011 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:51,011 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:52,806 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:53,012 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:53,835 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:45:55,012 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:55,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:45:55,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:45:55,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:45:57,012 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:45:57,872 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:45:59,013 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:01,014 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:02,984 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:03,014 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:05,015 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:07,015 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:08,198 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:09,016 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:10,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:46:10,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:46:10,018 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:46:11,016 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:13,016 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:13,230 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:15,017 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:17,017 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:18,296 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:19,018 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:21,018 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:23,019 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:23,363 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:23,836 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:46:25,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:46:25,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:46:25,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:46:25,021 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:27,021 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:28,428 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:29,022 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:31,022 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:33,023 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:33,495 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:35,023 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:37,024 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:38,560 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:39,024 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:40,017 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:46:40,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:46:40,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:46:41,025 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:43,025 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:43,626 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:45,026 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:47,026 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:48,693 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:49,027 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:51,027 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:53,028 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:53,760 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:53,836 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:46:55,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:46:55,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:46:55,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:46:55,028 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:57,029 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:46:58,827 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:46:59,029 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:01,030 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:03,030 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:03,895 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:05,031 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:07,031 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:08,962 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:09,032 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:10,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:47:10,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:47:10,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:47:11,033 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:13,033 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:14,029 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:15,034 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:17,034 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:19,035 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:19,100 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:21,035 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:23,036 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:23,837 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:47:24,174 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:25,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:47:25,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:47:25,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:47:25,036 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:27,037 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:29,037 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:29,241 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:31,038 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:33,038 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:34,308 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:35,039 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:37,039 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:39,040 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:39,374 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:40,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:47:40,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:47:40,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:47:41,040 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:42,267 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:47:42,268 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:47:42,269 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:47:42,269 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:47:43,041 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:43,041 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:47:44,480 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:45,041 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:47,042 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:47,767 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:47:47,768 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:47:47,769 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:47:47,769 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:47:48,042 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:47:49,042 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:49,574 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:51,043 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:53,043 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:53,837 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:47:54,638 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:47:55,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:47:55,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:47:55,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:47:55,044 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:57,044 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:59,045 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:47:59,702 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:01,045 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:03,046 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:04,770 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:05,046 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:07,047 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:09,047 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:09,835 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:10,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:48:10,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:48:10,019 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:48:11,048 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:13,048 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:14,903 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:15,049 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:17,050 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:19,051 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:19,973 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:21,052 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:23,052 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:23,838 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:48:25,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:48:25,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:25,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:48:25,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:48:25,052 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:27,053 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:29,053 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:30,147 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:31,054 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:33,054 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:35,055 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:35,228 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:37,055 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:39,056 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:40,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:48:40,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:48:40,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:48:40,290 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:41,056 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:43,057 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:45,057 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:45,361 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:47,058 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:49,058 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:50,427 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:51,059 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:53,059 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:53,839 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:48:55,018 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:48:55,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:48:55,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:48:55,060 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:55,495 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:48:57,060 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:48:59,061 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:00,588 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:01,061 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:03,062 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:05,062 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:05,655 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:07,063 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:09,063 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:10,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:49:10,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:49:10,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:49:10,722 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:11,064 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:13,064 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:15,065 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:15,809 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:17,065 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:19,066 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:20,894 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:21,066 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:23,067 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:23,839 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:49:25,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:49:25,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:49:25,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:49:25,067 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:25,972 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:27,068 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:29,068 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:31,040 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:31,069 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:33,069 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:35,070 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:36,107 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:37,070 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:39,071 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:40,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:49:40,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:49:40,020 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:49:41,071 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:41,178 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:43,072 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:45,073 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:46,249 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:47,073 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:49,073 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:51,074 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:51,344 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:53,074 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:53,840 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:49:55,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:49:55,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:49:55,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:49:55,075 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:56,417 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:49:57,075 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:49:59,076 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:01,488 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:02,077 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:04,077 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:06,078 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:06,556 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:08,078 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:10,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:50:10,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:50:10,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:50:10,079 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:11,624 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:12,079 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:14,080 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:16,080 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:16,697 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:18,081 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:20,082 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:21,770 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:22,082 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:23,840 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:50:24,083 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:25,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:50:25,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:50:25,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:50:26,083 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:26,867 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:28,084 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:30,084 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:31,940 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:32,084 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:34,085 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:36,086 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:37,007 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:38,086 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:40,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:50:40,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:50:40,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:50:40,086 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:42,074 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:42,087 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:44,087 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:46,087 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:47,141 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:48,088 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:50,088 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:52,089 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:52,207 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:53,841 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:50:54,089 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:55,019 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:50:55,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:50:55,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:50:56,089 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:50:57,273 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:50:58,090 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:00,090 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:02,091 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:02,341 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:04,091 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:06,091 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:07,408 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:08,092 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:10,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:51:10,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:51:10,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:51:10,092 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:12,093 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:12,474 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:14,093 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:16,094 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:17,540 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:18,094 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:20,094 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:22,095 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:22,607 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:23,842 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:51:24,095 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:25,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:51:25,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:51:25,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:51:26,095 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:27,674 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:28,096 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:30,096 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:32,097 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:32,739 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:34,097 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:36,097 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:37,806 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:38,098 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:40,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:51:40,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:51:40,021 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:51:40,098 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:42,099 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:42,871 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:44,099 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:46,099 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:47,936 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:48,100 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:50,100 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:52,100 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:53,001 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:53,842 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:51:54,101 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:55,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:51:55,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:51:55,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:51:56,101 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:51:58,067 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:51:58,102 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:00,102 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:02,102 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:03,133 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:04,103 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:06,103 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:08,103 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:08,198 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:10,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:52:10,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:52:10,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:52:10,104 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:12,104 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:13,281 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:14,104 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:16,105 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:18,105 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:18,356 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:20,106 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:22,106 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:23,422 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:23,843 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:52:24,106 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:25,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:52:25,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:52:25,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:52:26,107 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:28,107 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:28,488 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:30,107 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:32,108 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:33,555 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:34,108 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:36,109 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:38,109 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:38,622 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:40,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:52:40,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:52:40,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:52:40,109 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:42,110 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:43,689 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:44,110 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:46,111 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:48,111 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:48,754 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:50,111 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:52,112 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:53,820 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:52:53,843 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:52:54,112 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:55,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:52:55,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:52:55,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:52:56,112 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:58,113 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:52:58,887 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:00,113 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:02,114 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:03,956 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:04,114 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:06,114 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:08,115 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:09,069 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:10,020 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:53:10,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:53:10,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:53:10,115 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:12,116 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:14,116 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:14,127 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:16,116 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:18,117 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:19,205 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:20,117 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:22,118 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:23,844 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:53:24,118 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:24,277 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:25,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:53:25,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:53:25,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:53:26,118 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:27,287 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:53:27,288 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:53:27,288 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:53:27,288 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:53:28,119 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:28,119 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:53:29,119 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:29,415 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:30,119 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:32,119 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:32,857 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:53:32,858 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:53:32,858 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:53:32,858 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:53:33,120 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:53:34,120 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:34,438 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:35,120 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:36,120 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:38,121 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:39,497 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:40,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:53:40,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:53:40,022 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:53:40,121 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:42,122 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:44,122 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:44,577 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:46,123 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:48,123 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:49,678 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:50,123 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:52,124 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:53,844 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:53:54,124 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:54,743 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:53:55,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:53:55,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:53:55,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:53:56,125 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:58,126 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:53:59,812 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:00,126 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:02,127 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:04,127 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:04,881 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:06,128 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:08,128 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:09,946 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:10,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:54:10,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:54:10,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:54:10,129 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:12,129 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:14,130 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:15,011 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:16,130 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:18,131 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:20,077 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:20,131 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:22,132 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:23,846 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:54:24,132 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:25,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:54:25,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:54:25,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:54:25,348 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:26,132 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:28,133 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:30,133 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:30,440 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:32,134 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:34,134 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:35,506 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:36,135 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:38,135 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:40,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:54:40,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:54:40,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:54:40,136 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:40,571 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:42,136 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:44,137 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:45,637 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:46,137 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:48,138 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:50,138 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:50,706 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:52,139 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:53,845 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:54:54,139 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:55,021 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:54:55,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:54:55,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:54:55,772 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:54:56,140 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:54:58,140 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:00,140 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:00,839 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:02,141 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:04,141 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:05,905 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:06,142 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:08,142 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:10,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:55:10,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:55:10,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:55:10,143 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:10,971 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:12,143 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:14,144 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:16,038 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:16,144 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:18,144 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:20,145 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:21,104 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:22,145 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:23,846 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:55:24,146 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:25,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:55:25,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:55:25,023 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:55:26,146 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:26,169 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:28,147 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:30,147 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:31,236 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:32,147 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:34,148 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:36,148 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:36,301 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:38,148 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:40,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:55:40,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:55:40,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:55:40,149 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:41,367 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:42,149 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:44,150 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:46,150 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:46,433 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:48,151 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:50,151 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:51,502 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:52,152 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:53,847 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:55:54,152 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:55,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:55:55,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:55:55,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:55:56,153 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:55:56,578 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:55:58,153 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:00,153 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:01,587 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:02,154 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:04,154 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:06,155 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:06,626 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:08,155 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:10,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:56:10,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:56:10,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:56:10,155 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:11,692 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:12,156 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:14,156 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:16,156 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:16,761 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:18,157 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:20,157 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:21,840 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:22,158 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:23,847 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:56:24,158 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:25,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:56:25,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:56:25,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:56:26,158 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:26,909 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:28,159 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:30,159 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:31,976 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:32,160 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:34,160 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:36,160 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:37,041 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:38,161 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:40,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:56:40,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:56:40,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:56:40,161 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:42,106 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:42,162 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:44,162 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:46,162 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:47,173 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:48,163 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:50,163 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:52,164 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:52,240 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:53,848 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:56:54,164 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:55,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:56:55,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:56:55,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:56:56,164 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:56:57,307 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:56:58,165 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:00,165 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:02,166 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:02,376 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:04,166 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:06,167 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:07,430 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:08,167 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:10,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:57:10,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:57:10,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:57:10,168 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:12,168 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:12,473 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:14,169 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:16,169 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:17,503 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:18,169 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:20,170 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:22,170 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:22,572 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:23,849 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:57:24,170 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:25,022 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:57:25,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:57:25,024 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:57:26,171 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:27,638 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:28,171 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:30,172 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:32,172 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:32,704 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:34,173 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:36,173 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:37,769 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:38,173 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:40,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:57:40,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:57:40,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:57:40,174 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:42,174 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:42,835 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:44,174 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:46,175 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:47,902 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:48,176 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:50,176 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:52,176 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:52,951 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:53,849 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:57:54,177 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:55,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:57:55,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:57:55,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:57:56,177 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:57:58,028 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:57:58,178 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:00,178 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:02,179 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:03,140 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:04,179 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:06,180 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:08,180 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:08,207 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:10,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:58:10,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:58:10,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:58:10,181 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:12,181 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:13,273 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:14,182 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:16,182 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:18,182 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:18,339 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:20,183 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:22,184 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:23,387 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:23,850 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:58:24,184 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:25,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:58:25,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:58:25,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:58:26,185 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:28,185 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:28,412 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:30,186 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:32,186 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:33,475 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:34,187 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:36,187 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:38,188 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:38,547 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:40,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:58:40,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:58:40,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:58:40,188 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:42,189 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:43,616 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:44,189 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:46,190 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:48,190 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:48,689 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:50,191 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:52,191 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:53,702 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:58:53,850 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:58:54,192 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:55,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:58:55,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:58:55,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:58:56,192 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:58,192 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:58:58,748 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:00,193 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:02,193 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:03,816 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:04,193 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:06,194 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:08,194 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:08,883 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:10,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:59:10,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:59:10,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:59:10,194 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:12,195 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:13,388 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:59:13,389 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:59:13,389 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:59:13,389 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:59:14,195 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:14,195 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:59:14,390 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:15,195 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:16,196 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:18,196 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:18,933 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: partial_history 2024-02-23 03:59:18,934 DEBUG SenderThread:116329 [sender.py:send():382] send: history 2024-02-23 03:59:18,934 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: summary_record 2024-02-23 03:59:18,934 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 03:59:19,196 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 03:59:19,591 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:20,196 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:21,197 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:22,197 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:23,851 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:59:24,197 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:24,666 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:25,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:59:25,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:59:25,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:59:26,198 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:28,198 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:29,738 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:30,199 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:32,199 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:34,200 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:34,806 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:36,200 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:38,201 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:39,825 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:40,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:59:40,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:59:40,025 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:59:40,201 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:42,202 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:44,202 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:44,866 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:46,203 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:48,203 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:49,931 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:50,204 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:52,204 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:53,851 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 03:59:54,205 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:54,997 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 03:59:55,023 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 03:59:55,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 03:59:55,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 03:59:55,817 WARNING FileStreamThread:116329 [file_stream.py:request_with_retry():668] requests_with_retry encountered retryable exception: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')). func: functools.partial(>, timeout=180.0), args: ('https://api.wandb.ai/files/ameer08/huggingface/b657btrg/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 69, 'content': ['{"system.proc.memory.availableMB": 50206.44, "system.memory": 21.57, "system.proc.memory.rssMB": 1860.34, "system.proc.memory.percent": 2.91, "system.cpu": 3.13, "system.cpu.0.cpu_percent": 13.3, "system.cpu.1.cpu_percent": 18.17, "system.cpu.2.cpu_percent": 4.55, "system.cpu.3.cpu_percent": 3.53, "system.cpu.4.cpu_percent": 1.53, "system.cpu.5.cpu_percent": 1.19, "system.cpu.6.cpu_percent": 4.35, "system.cpu.7.cpu_percent": 8.64, "system.cpu.8.cpu_percent": 94.4, "system.cpu.9.cpu_percent": 0.47, "system.cpu.10.cpu_percent": 2.07, "system.cpu.11.cpu_percent": 1.84, "system.cpu.12.cpu_percent": 3.31, "system.cpu.13.cpu_percent": 4.17, "system.cpu.14.cpu_percent": 10.05, "system.cpu.15.cpu_percent": 27.68, "system.cpu.16.cpu_percent": 1.93, "system.cpu.17.cpu_percent": 2.26, "system.cpu.18.cpu_percent": 2.23, "system.cpu.19.cpu_percent": 1.01, "system.cpu.20.cpu_percent": 1.23, "system.cpu.21.cpu_percent": 1.0, "system.cpu.22.cpu_percent": 1.27, "system.cpu.23.cpu_percent": 1.17, "system.cpu.24.cpu_percent": 0.83, "system.cpu.25.cpu_percent": 1.53, "system.cpu.26.cpu_percent": 1.27, "system.cpu.27.cpu_percent": 1.27, "system.cpu.28.cpu_percent": 1.07, "system.cpu.29.cpu_percent": 0.93, "system.cpu.30.cpu_percent": 0.97, "system.cpu.31.cpu_percent": 1.06, "system.proc.cpu.threads": 68, "system.network.sent": 56252907.4, "system.network.recv": 50977847.93, "system.disk.\\\\.usagePercent": 97.96, "system.disk.\\\\.usageGB": 1703.95, "system.disk.in": 126.53, "system.disk.out": 11708.95, "system.gpu.0.memoryAllocated": 61.6, "system.gpu.process.0.memoryAllocated": 61.6, "system.gpu.0.memoryAllocatedBytes": 15866291814.4, "system.gpu.process.0.memoryAllocatedBytes": 15866291814.4, "system.gpu.0.memory": 76.13, "system.gpu.process.0.memory": 76.13, "system.gpu.0.gpu": 95.8, "system.gpu.process.0.gpu": 95.8, "system.gpu.0.temp": 79.2, "system.gpu.process.0.temp": 79.2, "system.gpu.0.powerWatts": 400.2, "system.gpu.process.0.powerWatts": 400.2, "system.gpu.0.powerPercent": 88.93, "system.gpu.process.0.powerPercent": 88.93, "_wandb": true, "_timestamp": 1708640993.851158, "_runtime": 2130.86618}']}, 'output.log': {'offset': 19, 'content': ['ERROR 2024-02-22T22:29:55.810587 11%|█████████ | 308/2896 [00:36<04:57, 8.69it/s]\n']}}, 'dropped': 0}} 2024-02-23 03:59:56,205 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 03:59:58,206 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:00,064 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:00,206 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:02,207 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:04,207 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:05,130 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:06,207 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:08,208 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:10,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:00:10,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:00:10,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:00:10,208 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:10,393 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:12,209 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:14,209 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:15,492 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:16,210 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:18,210 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:20,211 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:20,558 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:22,211 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:23,852 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:00:24,212 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:25,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:00:25,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:00:25,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:00:25,624 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:26,212 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:28,213 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:30,213 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:30,690 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:32,214 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:34,214 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:35,757 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:36,215 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:38,215 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:40,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:00:40,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:00:40,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:00:40,216 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:40,776 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:42,216 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:44,217 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:45,851 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:46,217 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:48,218 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:50,218 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:50,877 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:52,219 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:53,852 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:00:54,219 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:55,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:00:55,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:00:55,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:00:55,944 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:00:56,220 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:00:58,220 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:00,221 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:01,011 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:02,221 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:04,222 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:06,080 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:06,222 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:08,222 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:10,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:01:10,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:01:10,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:01:10,223 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:11,145 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:12,223 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:14,224 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:16,211 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:16,224 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:18,225 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:20,225 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:21,275 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:22,226 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:23,853 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:01:24,226 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:25,024 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:01:25,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:01:25,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:01:26,227 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:26,341 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:28,227 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:30,228 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:31,405 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:32,228 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:34,229 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:36,229 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:36,471 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:38,229 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:40,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:01:40,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:01:40,026 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:01:40,230 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:41,540 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:42,231 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:44,231 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:46,232 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:46,653 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:48,232 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:50,233 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:51,700 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:52,233 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:53,853 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:01:54,234 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:55,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:01:55,026 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:01:55,027 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:01:56,234 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:01:56,794 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:01:58,235 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:00,235 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:01,860 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:02,236 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:04,236 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:06,237 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:06,925 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:08,237 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:10,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:02:10,027 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:02:10,027 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:02:10,238 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:11,990 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:12,238 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:14,239 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:16,239 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:17,057 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:18,240 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:20,240 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:22,131 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:22,241 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:23,855 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:02:24,241 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:25,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:02:25,027 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:02:25,027 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:02:26,242 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:27,203 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:28,242 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:30,243 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:32,243 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:32,272 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:34,244 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:36,244 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:37,340 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:38,245 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:40,025 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-23 04:02:40,027 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: stop_status 2024-02-23 04:02:40,027 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: stop_status 2024-02-23 04:02:40,245 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:42,246 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:42,407 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:44,246 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:46,247 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:46,388 DEBUG SenderThread:116329 [sender.py:send():382] send: exit 2024-02-23 04:02:46,388 INFO SenderThread:116329 [sender.py:send_exit():589] handling exit code: 255 2024-02-23 04:02:46,388 INFO SenderThread:116329 [sender.py:send_exit():591] handling runtime: 2302 2024-02-23 04:02:46,388 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 04:02:46,389 INFO SenderThread:116329 [sender.py:send_exit():597] send defer 2024-02-23 04:02:46,389 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,389 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 0 2024-02-23 04:02:46,389 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,389 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 0 2024-02-23 04:02:46,389 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 1 2024-02-23 04:02:46,389 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,389 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 1 2024-02-23 04:02:46,389 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,389 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 1 2024-02-23 04:02:46,389 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 2 2024-02-23 04:02:46,389 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,389 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 2 2024-02-23 04:02:46,389 INFO HandlerThread:116329 [system_monitor.py:finish():203] Stopping system monitor 2024-02-23 04:02:46,389 DEBUG SystemMonitor:116329 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-23 04:02:46,390 DEBUG SystemMonitor:116329 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-23 04:02:46,390 INFO HandlerThread:116329 [interfaces.py:finish():202] Joined cpu monitor 2024-02-23 04:02:46,391 INFO HandlerThread:116329 [interfaces.py:finish():202] Joined disk monitor 2024-02-23 04:02:46,415 INFO HandlerThread:116329 [interfaces.py:finish():202] Joined gpu monitor 2024-02-23 04:02:46,415 INFO HandlerThread:116329 [interfaces.py:finish():202] Joined memory monitor 2024-02-23 04:02:46,415 INFO HandlerThread:116329 [interfaces.py:finish():202] Joined network monitor 2024-02-23 04:02:46,415 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,415 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 2 2024-02-23 04:02:46,415 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 3 2024-02-23 04:02:46,415 DEBUG SenderThread:116329 [sender.py:send():382] send: stats 2024-02-23 04:02:46,416 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,416 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 3 2024-02-23 04:02:46,416 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,416 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 3 2024-02-23 04:02:46,416 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 4 2024-02-23 04:02:46,416 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,416 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 4 2024-02-23 04:02:46,416 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,416 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 4 2024-02-23 04:02:46,416 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 5 2024-02-23 04:02:46,416 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,416 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 5 2024-02-23 04:02:46,417 DEBUG SenderThread:116329 [sender.py:send():382] send: summary 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-23 04:02:46,417 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 5 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 6 2024-02-23 04:02:46,417 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,417 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 6 2024-02-23 04:02:46,417 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 6 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 7 2024-02-23 04:02:46,417 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: status_report 2024-02-23 04:02:46,417 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:46,417 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 7 2024-02-23 04:02:46,417 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:46,417 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 7 2024-02-23 04:02:47,247 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 04:02:47,388 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: poll_exit 2024-02-23 04:02:48,247 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:49,068 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 8 2024-02-23 04:02:49,069 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: poll_exit 2024-02-23 04:02:49,069 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:49,070 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 8 2024-02-23 04:02:49,070 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:49,071 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 8 2024-02-23 04:02:49,071 INFO SenderThread:116329 [job_builder.py:build():296] Attempting to build job artifact 2024-02-23 04:02:49,073 INFO SenderThread:116329 [job_builder.py:_get_source_type():437] no source found 2024-02-23 04:02:49,073 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 9 2024-02-23 04:02:49,073 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:49,073 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 9 2024-02-23 04:02:49,074 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:49,074 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 9 2024-02-23 04:02:49,074 INFO SenderThread:116329 [dir_watcher.py:finish():358] shutting down directory watcher 2024-02-23 04:02:49,247 INFO Thread-12 :116329 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():388] scan: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/requirements.txt requirements.txt 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-metadata.json wandb-metadata.json 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log output.log 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/conda-environment.yaml conda-environment.yaml 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json wandb-summary.json 2024-02-23 04:02:49,248 INFO SenderThread:116329 [dir_watcher.py:finish():402] scan save: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/config.yaml config.yaml 2024-02-23 04:02:49,248 INFO SenderThread:116329 [sender.py:transition_state():617] send defer: 10 2024-02-23 04:02:49,253 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: defer 2024-02-23 04:02:49,253 INFO HandlerThread:116329 [handler.py:handle_request_defer():172] handle defer: 10 2024-02-23 04:02:49,254 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: defer 2024-02-23 04:02:49,254 INFO SenderThread:116329 [sender.py:send_request_defer():613] handle sender defer: 10 2024-02-23 04:02:49,254 INFO SenderThread:116329 [file_pusher.py:finish():172] shutting down file pusher 2024-02-23 04:02:49,388 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: poll_exit 2024-02-23 04:02:49,388 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: poll_exit 2024-02-23 04:02:50,102 INFO wandb-upload_0:116329 [upload_job.py:push():131] Uploaded file /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/output.log 2024-02-23 04:02:50,128 INFO wandb-upload_2:116329 [upload_job.py:push():131] Uploaded file /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/conda-environment.yaml 2024-02-23 04:02:50,130 INFO wandb-upload_3:116329 [upload_job.py:push():131] Uploaded file /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/wandb-summary.json 2024-02-23 04:02:50,132 INFO wandb-upload_4:116329 [upload_job.py:push():131] Uploaded file /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/config.yaml 2024-02-23 04:02:50,364 WARNING StreamThr :116329 [internal.py:is_dead():414] Internal process exiting, parent pid 116027 disappeared 2024-02-23 04:02:50,364 ERROR StreamThr :116329 [internal.py:wandb_internal():152] Internal process shutdown. 2024-02-23 04:02:50,388 DEBUG HandlerThread:116329 [handler.py:handle_request():146] handle_request: poll_exit 2024-02-23 04:02:50,389 INFO HandlerThread:116329 [handler.py:finish():869] shutting down handler 2024-02-23 04:02:50,389 INFO WriterThread:116329 [datastore.py:close():296] close: /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/run-b657btrg.wandb 2024-02-23 04:02:50,389 DEBUG SenderThread:116329 [sender.py:send_request():409] send_request: poll_exit 2024-02-23 04:02:50,389 INFO SenderThread:116329 [sender.py:finish():1572] shutting down sender 2024-02-23 04:02:50,389 INFO SenderThread:116329 [file_pusher.py:finish():172] shutting down file pusher 2024-02-23 04:02:50,389 INFO SenderThread:116329 [file_pusher.py:join():178] waiting for file pusher 2024-02-23 04:02:51,034 INFO wandb-upload_1:116329 [upload_job.py:push():131] Uploaded file /home/rnd/Documents/Ameer/gemma/wandb/run-20240223_032422-b657btrg/files/requirements.txt 2024-02-23 04:02:51,235 INFO Thread-11 :116329 [sender.py:transition_state():617] send defer: 11 2024-02-23 04:02:51,390 INFO SenderThread:116329 [file_stream.py:finish():595] file stream finish called 2024-02-23 04:02:52,525 INFO SenderThread:116329 [file_stream.py:finish():599] file stream finish is done